You did better than previously that's good But he does rattle you, doesn't he?
Some questions: When do you feel the rise in you - what's the specific trigger? If you can identify that you can maybe say to your Self next time - here it comes, I'm feeling xxx but this time I will 180 the reaction. What would he least expect you to do? Go silent? No explanation one word answers? Plan something for the next time.
Do you think he likes getting reactions? The way you describe the interaction it seems he goads you for them.
If your boundaries are going to work for your own self-respect, I think you will need to walk away when the inner tension starts and definitely not "give" in on seeing the kids in future. Believe me, MKB, I know it's easier said than done