You basically gave him PERMISSION to have an affair, as has been pointed out to you above.
That being said, there's nothing written that says you can't change your mind.
"I have decided that I'm no longer willing to live in an open marriage, and I'm no longer willing to lie to cover up your affair" would be a great place to start.
