Punchy, I know what you mean. When my W was away it was just enjoying the kids and getting through the day, without all the added stress.
Wish I knew what advice to offer about the confrontation regarding the OM, but I'm in that boat myself. I've avoided that talk, even though I've planned to have it several times. I didn't know absolutely what had been happening - so I knew W would have denied it and I'd be back to where I was, but looking bad with a really POed W feeling more justified.
In retrospect I'm glad I've waited, because I'm now in a much better place to have that conversation. I've attempted to collect some information so I can know what's really going on.