Jenna Sorry for the lack of response. Him threatening to leave again was him grasping at whatever emotional leverage he had to get you to back off.
Yes it is true that you need to let go of OW, otherwise you'll just drive him to her or another one again. If you treat him like he is still cheating, well then he has nothing to lose. (people are really good at crazy rationalization like that)
That being said during piecing it is important that the returning spouse learns that threatening to leave the relationship is a very dirty fighting technique. If you let it slide, and more importantly if he gets his way you will see him use this again and again.
Next time just call his bluff. It's incredibly painful to do so, but you don't want him threatening to leave because you bought the wrong type of soda. (it's a ridiculous example, but it can get pretty crazy at times)
Besides who wants to stay with someone who threatens to leave at the slightest sign of trouble.
One more thing, I'm a little concerned about him (in your words) being a pushover. It may not be very obvious right now, but you are slowly losing respect for him, this is a sure fire way for a woman to lose attraction for their partner. IMHO this is why a lot of guys get cheated on.
Of course you yourself can't get him to do this, but maybe talking to your therapist about it in private.