Women are not the only creatures on God's earth that have a right to change their mind.
Thus:
"Wife, I realize that I told you that I had no problem with you talking to this other man, but it's clearly gotten beyond that -- even if only by him -- and I've decided that I've changed my mind. It's not appropriate, and I'm not going to live in a marriage with a third person in it."