In fact one time H said that his family "warned" him about me so I don't think they've liked me much from the beginning. But if I stay away, that just reinforces all of it. So I don't know what to do.
LOL...you could "warn" anyone about their choice in a spouse and rest assured, your odds of being right are better than 50%!
If H invites you, then I would go and act as if. Enjoy every moment....enjoy the time with your kids and your extended family. Don't worry about what some of them might think about you....be true to yourself! Don't let them control what you do.