Well, it seems reasonable to me that as soon as he has better living arrangements, the current 'visit them here' deal is off. It sounds as though he could take them to his new place (if friend is happy, or is out) to his parents, or take them out somewhere. You could always let him know that having him there visiting with the kids in your home isn't working for you and He'll need to make other arrangements. And (frankly) what other arrangements he chooses to make are up to him and needn't concern you as long as the kids are safe and well looked after.

JMHO of course, but I don't think there's any need to let him have things every which way he wants at the expense of your convenience and wellbeing?

T 13 M 7
Me 48 H 46
SS 15
BD 7.14 PA
D final 5.16 (H filed)

We receive & we lose, and must try to achieve gratitude & embrace with whole hearts whatever of life that remains after the losses - Dubus