Vise and Broke gave really good ideas for handling the kids. I was always kind of a single mom. My XH used to travel a lot. The way it worked for me was really a good communication.
Working as a team always worked well. Till today my kids are like that. It was always about agreeing what to do and always keep an eye on each other.
My kids learned very early their full name, address and phone number. It can be overwhelming at first, but the more you do, the more they get use to it.
I agree with Tx, at first your H would come and go, but now you know there is an A going on. I really believe that at some point you two need to develop some kind of schedule with the kids.
It's not even about you and your looks anymore. This is not good for the kids that he just pop up when he feels like. They may not show now, but this can create deep wounds that will show up when they are teenagers.
Just because they are very young does not mean they are not internalizing all what is going on around them. And I know, it is asking one more thing for you to take care after. It's very unfair, I sometimes feel the same way.
Why should be me to think about all this. But, it is what it is for now and you are the only adult that can take care after then. You potato head H does not even realize what he is doing for his own kids.
You know some of his schedule, so put a plan on a paper and ask him to look over and talk to you if he needs to change anything. You will not ask for his permission, or his approval on this. You need to tell him this is the right thing for your kids.
This is not going to play against you. It will be a boundary that needs to be respected and this is also part of the DB process. Like Tx said, he is the one that checked out of the team, he needs to find out about the consequences.