"We re going to the movies/fun dinner /bounce house /whatever. We'll see you later, bye!" . Can you drop the little one with your mom and take your son to do something fun?
Be someone only a fool would leave. Would a fool leave someone who said, "I can't be around you when you are getting ready to go out because it hurts me too much "? NO.