then he put his spin on it, saying that I was wrong to believe that he was with her all the time.
Of course he's not with her ALL the time. (When he said it, he wasn't with her, he was with you. Duh.) Classic misdirection. It would be so tempting to ask, "Am I wrong to believe that you are EVER with her?" Or how about, "So, should I have the kids call you at your mom's number from now on?"