Wow, that sounds pretty hard. You dealt with it assertively and kindly I think. And buying a bigger house so he can have space? Ummmm, lol.
Do you think there might have been something he was actually guilty about with the phone for him to react like that? I don't think it's an unreasonable question actually, particularly in light of his EA.
Well done, you handled it well. I hope he feels your absence. And I hope you have a lovely time!