It sounds like he was trying to push your buttons on purpose because you were not giving him the answer he wanted.
ignore it, he is acting like he's 2, and he's not going to change.
I would do what Mc suggested if you really don't want to go. I wouldn't leave my children either on Mom's day, especially since all that is going on with him.
If you tell him that he cannot take her without you, and he doesn't want you to go, what do you think he would do??
I would say let him make that comment about his family, then I would say to him "would you like me to tell them why she isn't going and why Im not going?, that may shut him up.
I don't think its a good idea for you to be in a car with him that long... what state are you in? I want to check some flights for you.
me: 37 H: 44 Married for 18 years this june S7 S3 porn issues, and much more... since 7/06
Happiness can be found, even in the darkest of times, if one only remembers to turn on the light.