I would say, "Sorry you feel that way. i thought we could have a nice time together. But if you don't want me to come, then I'll do something else." And then I would go on a nice vacation by myself, charging it to him, of course.