I understand completely what you're saying. I'm just not sure if I have the strength to stick with that right now. I know that once I lay that out, I CAN'T go back on that statement.
Look, it's either what I call a "Boundary of Personal Integrity" with you or it's not. I can't convince you that you really SHOULD have as a dealbreaker boundary "I won't be in your life as your husband as long as you are having an affair with another man." All I can tell you is what I would do, and also point out to you that such a stance (tolerating it) is very likely KILLING ATTRACTION between your wife and you.
I would say that to ANY man, but all the more so considering your wife's very clear "Why didn't you fight for me to stay?" comments.
It's entirely your choice, but I personally have never seen the approach you're currently taking work. Even those that take a "softer" approach than I do on the whole flirting/be-their-friend/pursuing thing, have taken a very hard stance on the "no-contact" piece of it (see MulesQB and Denver2010 for just two great examples), and often couple it even with a strong legal stance.