I agree with you that the h/w relationship needs to come first, but what you are doing is putting him in an either/or situation. HE doesn't view it that way, YOU do.
It's either me first or them first.
If I have misinterpreted this, please correct me.
I believe what everyone here has been suggesting to you is an alternate route to the either/or, and you and your h can both win.
It could be that he doesn't include you because you have made it very clear you view his family as an infringement on your life -- so he may just assume you don't want to participate.
This you could clear up very easily by asking him.