They very likely had a fight and she threw him out but they got back together again sometime after that. That's my guess anyhow...
The phone reocrd thing is brilliant... bravo...
If he complains about you exposing his affair give him this line :
"As long as you are cheating - I am talking about it. You seem to think this is an open marriage so I say its also a free country and I know my rights and one of them is to speak my mind."