You and I both know he would be happier with you and his family. But he now probably still *thinks* he was UNHAPPY with you and may still *think* happiness is with her. The truth is he's probably unsure where happiness is. Just wish it to him... because that's what REAL love is. When you really love someone you let them go. Keeping someone who is unhappy is selfish! So love unconditionally and WISH him happiness. That will make him focus on trying to figure out where that is rather than focusing on the problems with you. Because it's only IF he decides completely on his own that happiness is with you and not elsewhere, then he'll be ready to be back and there will be a much lower chance of this ever happening again. If he comes back wavering that wouldn't be so great.
And yes, if something is uncomfortable for you... like you don't want to be around her... you have a right to say... sorry, I'm busy... but I hope you are well (and boy if you wish her well that would certainly make you appear like one unique "uncatty" woman!!! Even if the real truth is you hope all her hair falls out and she gains 200lbs!!!).
Oh, here's the other thing I just remembered that I wanted to mention earlier.... Guilt, condemning, blame, anger will only make it more impossible for him to "come back" if he begins to question his decisions. No one wants to return to that. The best way to "leave the door open" is with friendship...
And if you have a nice friendship with him... what do you think she'll do? Hummmm.... tables may start to turn....
There is no arriving, ever. It is all a continual becoming.