I would say, "Sorry you feel that way. i thought we could have a nice time together. But if you don't want me to come, then I'll do something else." And then I would go on a nice vacation by myself, charging it to him, of course.
Ooops, I was writing my post before I saw yours Sara.
I will take a vacation for myself after I get moved out.
