Ultimatley you have to decide to let go. I left the OW a few times during our affair, but it was because I felt I had to. After the reality of what I was doing and about to do set in, I WANTED to do it.
I'm not sure what will get you to that point, but I hope you do or many people will suffer.