You aren't piecing, at least not what I would consider piecing. I consider it two people committed to making their marriage work and repairing the damage. He isn't on that page. He's just as liable to leave as stay and just as likely to cheat with OW some more as not. You have justifiable trust issues. He shouldn't be trusted. I personally don't even know why you let him back into the house. Your self-esteem has to be in the gutter to take this kind of b.s.
My advice, and you'll probably get many dissenting opinions, is that unless he's completely done with the OW, I would ask him to leave. Yeah, he'll possibly run back to her, but then again, he might suddenly wake up and smell the coffee. He knows he doesn't have to work to regain your trust because you're so desperate to have him and needy. That's the only way to describe it. If I had a woman that needed me so badly that she walked on eggshells after I screwed up, I would have very little respect for her and feel I could do whatever. You have got to quit being his doormat. I predict that even if you somehow work this out, you'll build up enough resentment that you won't be able to do this anymore. He needs to be apologetic and willing to assuage your fears; not put you off with anger. He needs to kiss your a$$; so why is it that you are kissing his?
In the end, it's not the years in your life that count. It's the life in your years. Abraham Lincoln
It is hard to fail, but it is worse never to have tried to succeed. Theodore Roosevelt