If it were me, I'd respond with "I'd love to talk about our marraige, but not as long as there's a third person in it. End your affair (alt.: "contact with OW"), and we can discuss. And I'd hope that something as important as our family's future, you would want to discuss completely sober."
But that's just me.
Even if you don't want to OFFENSE-ively throw out ultimatums, there's certainly nothing wrong with RESPONDING with what your boundaries are (no contact with OW) if HE decides he wants to "re-assess" the marriage. In fact, it's entirely appropriate.
