H has started an affair 2

[Painter]

I'd ask in a bewildered voice: "The kids and I went by to treat you to lunch so you could see them - but you weren't there??"

It might not be a great idea but I probably couldn't help myself...

On the other hand, they're his kids. He doesn't "watch them" for you. They are a shared responsibility. Maybe you need to suggest a separation agreement that includes a custody schedule, if he's going to bail out on them.

Or - you could tell him that then you'll ask the in-laws.

DISCLAIMER: I have no idea if any of this is DB-approved. It's just what I would consider.