Contact is great progress! Remember how far you've come!
I don't think it was a mistake to invite him to the party, just move on. Try to detach and not be disappointed when he doesn't come to things. This is a big part of dealing with a skittish spouse. Not that I'm great at it, but here are some tips. Act confident and nonchalant, like it's no big deal whether he comes or not. If he sense desperation or disappointment it will scare him. Keep things low-key. I know this is hard, but you're doing great.
Maybe text him sometime soon about the holidays. I'm sure he's thinking about it as well.
Me: 30 W: 28 T 8, M 6 S: 7-27-2007 W filed (again) 3-2011 Served 8-2011 Responded, now dark "I have decided to stick with love. Hate is too great a burden to bear" MLK