Like everyone else has said. I would say to H that you have made plans and that it would be difficult for you to cancel it. My ex tried it on me too, and like you I was worried but I reasoned myself by saying that even if ex is in Lalaland I don't think he'd put his own kids at risk. He didn't like it but I felt that has showed him that I will no longer be a fixer.
