You stopped the conversation before it spiraled out of control. Lord knows getting into a game of he said she said where you both end up mad or crying or both in front of each other isn't a better ending.
You can't fix him. If he's upset, it's just as likely not about you he's breaking up his family, he damn well should be upset, ya know?
Go for a walk if you need? Or a short drive? Sometimes fresh air can just help you breathe a bit.