//I feel silly inviting myself along and getting in the middle of it//
I don't think this is silly. This is your family and you are still married. Of course you want to be a part of the vacation. He is a big boy, if he doesn't want you there, he can say so.
