I also did everything wrong.

I don't think so. I like how calm you stayed when he became emotional. It's also very good that you've stated your boundaries. Enjoy your weekend with the kids!