If it's to see the kids, tell him to ask them. If they're okay with it, you'll be happy to clear out for a few hours so that he can spend some time with them. If he asks why you're not staying, just say that you're not interested in a friendship with him. As long as he's with OW, he won't be with you. Sounds like he's trying to set up a cake-eating situation.
I'd let him know they're busy on Thursday night and he should coordinate it with them. If it doesn't work out for Thursday, he's welcome to set up a dinner night with them on a different night. You just need to know which one, so that you can make plans.
