Why do you have to do dinner with him? Why not tell him you don't want to do dinner with him. Just let him bring the kids home when he's had his fake father's day.