Okay, I realize this is all overwhelming right now. Let's slow down and breath.
I empathize with your anxiety of pushing him further away. Been there, done that. Let's rewind a little bit. In your first response to me, you said that you needed to learn how to communicate your needs better. That takes us to what we are dealing with at this point. You are paralyzed by the fear of pushing further away...or completely away altogether. Let's call it the LBS fog of fear.
First, you will need to overcome this fear. It takes times. This is something I had to overcome myself. Even during my marriage, before the mess. Instead of saying no to things, I would go along with my W to avoid disagreements. Later they would boil over and I would become upset. Now, I will say no to certain things. The right delivery makes a world of difference.
I tell you this, because I think you may have been the same way in your marriage based on your responses.
So to answer your questions. It isn't your job to change him. People change, only when they choose to change. Now, you can influence change on him, by first changing yourself. Like you already said, you need to learn how to communicate your needs to him. That should be your first assignment.
You asked, what do you say? As I mentioned above, delivery makes a world of difference. You can always communicate in a loving manner without being rude or defensive. When you are ready, you can say something like, "H, you know that I love you and want to work things out, however, I cannot continue with a third person in our situation. We need to operate independent of one another, with the exception of the kids or house(whatever else requires contact) only." Then walk away or if it is on the phone, let him know that you have to go. Don't wait for his response. Don't get caught up in his anger. That's not for you to worry about. You own your part and address your issues.
You have to be ready to do that. It has to be genuine. It isn't a trick or game to get him back. It took me a long time to get there. I'd guess about 14 months. I confirmed the affair 12-13 months in.
Also, journal your feelings, thoughts and things you might say. It helped me to remember what I planned on saying and with my delivery.
Me:45 ExW:48 M:04/97 3 Bombs & 2 ReCons 1st BD 11/10 D Finalized 4/20 D-16 S-14 Going in one more round when you don't think you can. That's what makes all the difference in life.~Rocky Balboa