i say...if he is paying for it...

go on the damn trip
have a blast on his dime
make memories with your son
and
if your husband is in some of them

great for him

if he isn't

well then...too bad for him

go with the idea that this is for you and your son

and

not your FAMILY

don't go if you are dreaming of a family vacation
that isn't what this is