If you just step back, let him do his stupid thing, and let the affair burn itself out, it could work out to your advantage in the end.
That is what I meant when I said maybe after OW is back for a little bit. Yes, he will have to avoid her...but it would be better if it was his choice rather than a demand from you.