Tired of trying to be his friend while he is still contacting her.
Maybe, maybe, he is contacting her like a friend not as an intimate R. I remember telling my W about a converstation I had with another woman at work, and she got jealous. Well, I found that only made me feel untrusted - and what is a marriage without trust.
JUST IDEAS from a semi-WAH point of view... - make sure not to flip flop - don't close doors you might want open later (if you want him in your life, let him know the door is open) - don't beg (control your emotions) - do not go "dark" and cut off contact. If I was alone but had a R over email with someone and my spouse didn't want to talk, well...guess who does? - do not give ultimatums unless you really are ready to follow through
I noticed that you asked him to get out, but then you said he came back a few times for weird reasons. Look for those as opportunities for him to move back in. Yes, the contact with the OW needs to end. But he needs to choose that. If he's just talking, is that OK with you that he might want to only end it if your R gets going again?