You think it would be best to not let him come? Part of me says not to, but part of me wonders if he might be waking up a little.
If I were you, I would say that I prefer to go alone (no more explanation needed than that). He would be "cake-eating" if he went to the party. He would be enjoying the good part of his marriage without fulfilling his responsibities. You need to make it clear that you will not tolerate it. This will help him "wake up".
