I'm finding that having no contact with him makes me feel distant from him. Won't he be feeling the same thing? Now the OW can fill the space that I am leaving.
That's the idea. As long as he's getting some of his emotional and physical needs met by you, and some by his OW, then he's never going to change the path he is on. Letting illicit affair partners be together, warts and all, can often take the mystique off of the relationship. So that's part of it.
But he also needs to feel like he's losing YOU. I've been at this here on this forum (as Starsky and previously two other usernames, going back over 11 years and over 25,000 posts, studying affairs) and I have never YET seen a marriage saved when there was infidelity involved, where the cheating spouse didn't have a credible fear that they were beginning to lose their betrayed spouse.
Until he credibly fears that he might lose you, you'll never get him back.
Now, don't get me wrong -- people caught up in affairs often don't CARE if they lose you, so it's not guaranteed to work. But I can guarantee you it WON'T work if he thinks he's just got you sitting there like a puppy dog as his "plan B" if things don't work out with his OW.
People value most that which is difficult to achieve, and us men especially like the the thrill of the hunt.