An addict is an addict. You aren't to blame for another's addictions. We all have choices in how we handle out problems.
It's not about you or the OW being enough, it has nothing to do with people or feelings about people. He probably doesn't see either of you as people, but rather as a means to feed his addiction.
Unless he sees it as a problem and gets help, this behavior will continue.
Do you want to you live a life like that?
Me 57/H 58 M36 S 2.5yrs R 12/13
Let me give up the need to know why things happen as they do. I will never know and constant wondering is constant suffering. Caroline Myss
