((((((Mindfull))))))
In my experience, with kids fairly similar in age, going between homes doesn't seem to bother them a bit. I think they'd get more confused by the mixed message of him popping in and out whenever the mood struck him.

Isn't it interesting... in his plan, he gets to have his alternate life, and yet he still gets to control yours! I think not! He needs to have time (not necessarily equal) with the kids, when he is responsible for the kids, on his turf. And you can have a life on yours,