Oh, I'm tempted to tell him no he can't go with us on Saturday (hopefully to make him see what he's missing out on), should I?
Definitely. If he asks why, tell him that you feel it would be uncomfortable for everyone.
None of my friends know about this yet. He would go and act normally. As if things were fine. I just can't figure out why he wants to go. What is he getting out of it?