Let him see the kids, do you know how long he's going to be seeing them? Maybe a couple of hours? Use that to your advantage ;-)
When he comes over, make sure you are dressed appearing to go out, put on some hot clothing, do the hair & make up thing, put on some perfume, use the "fancy" purse, not the bag you usually carry with you, put on some nice shoes and appear to be going out.
Tell him you'll be back in an hour but come back in 3 hours.
He will call, text, when you don't make it back on time, take your time replying back, you can reply with "got held up at a friends, I should be home soon".
And that's it.
VOILA!
Mystery, suspense, drama and on top of that you had a baby sitter for the evening and you didn't even know it. Go to the movies or something, by yourself or with a friend, just do it.
When you arrive in the door of your home, pick up your phone as if you're talking to someone and tell them "sorry I had to cut our evening short, kids & babysitters, I tell ya LOL! Yes I will find a more RELIABLE babysitter next time, I'm home now can I call you in a bit? Sounds good... bye"
Your H will ask you who you were talking to, and you will reply, "just a friend", and he may ask man or woman, you will reply, "does it matter, its just a friend, stop asking so many questions, you're here to see the kids not asking me 20 questions" - give him attitude when you say this.