FWIW, I would vote on going - even if you only go for your H and your children. And only if you can resolve to go and relax, and not to stress if they aren't welcoming to you. So that's what I would do...but hey, I've been known to be wrong and to go a bit too far in accomodating others!
But either way, I think you've explained yourself well and you deserve to have a wonderful holiday!
