It's lovely of his family to invite you, but if it was me, I wouldn't go. It's going to be mega awkward and there's potential for drama of some kind or another. Do you have family of your own that you can visit, or good friends who can invite you to their holiday? I'm betting you would have more fun with a friend's family than being awkward around H.
Added bonus: if you're not there his family members are free to spend the holiday telling H what an idiot he is
