Perhaps just like it took for me as a WAW he has to really feel like I am done and I don't think he gets that yet.
I can't imagine how being a pushover could help your situation. Your failure to disapprove is essentially approval. If it were me, I would basically say, "I'm done. I know about the affair and I'm through with you." And leave it at that.
It sounds like you learned most by spying? Is that correct? Are you saying you haven't even told him you know yet? If he knows you know, and is doing this stuff in front of your face, then he doesn't care you know. That's worse. If he just thinks he's being sneaky, then it's different. You responding with a firm barrier (ie you will not be with him while he's cheating)may be enough to shake him up into pursuing you instead.
There are different approaches. You could do the DB ignore the affair stuff (and let your self-esteem continue to circle the drain) or you could set some boundaries, which basically is you won't tolerate cheating.
You cannot be lonely if you like the person you're alone with. Dr. Wayne Dyer
