oh ok, sorry I saw your usage of things got ugly, u meant only by her then, my apologies.
Seems like any R talk, even when she brings it up in a negative way is too much. If she goes there, maybe change the subject, she's still rewriting marital history. This is usually a time you back off and stay away best as possible. This is a time that nothing you say can help, only make things worse, so try to not let it go there.
You said it yourself, when your not home, the OM is coming by. Its out of your control, and you'll never get the solution you desire. It sounds like the guy would drive her crazy in a week or so if given the chance, maybe that's your play. Try to get yourself out of the house more, until late, when u come home and go right to bed. I promise she'll wonder where you've been, and eventually might even ask. That mystery thing again.
Backing off, staying away from the house as much as possible, and when you do come in, have a smile on your face. Say hi and go to your room. let the process do its work for you. It does work given the time. On weekends, get the kids up early and take them to breakfast, then the park or something, get some kites, make a picnic out of it. Stay back and away as much as possible and trust in it all. Its gonna take some time, don't get impatient with it. He cant score brownie points if your being a good father, he just thinks he can.