Him saying he doesn't want to be with you is based on emotion, he's infatuated with the fantasy of OW, it may change over time, or it may not, but nothing you can do about it so focus stays on you.
If you're going out, and he wants to stay with the kids, absolutely do that, he's their father, don't block that.
As for him wanting to see them, tell him you want him to do that too, but a schedule needs to be set up, so it's planned. Don't lie about having plans, instead just say the truth, I'm going out. I would let him know he needs to tell you when he will be taking them out to do things or staying at the house so you can plan accordingly. If your up to staying in the house without being confrontational, then that's ok, but if it hurts to much, just go out. With a schedule, you'll be able to plan with friends and family to do something or visit them.
M - 9 1/2 years 5/5/16 - Bomb drop - 3 week EA 10/31/16 - We sold house 01/10/18 - D Finalized