The one thing about affairs is you have to let them play out. You can not do anything to stop your H. They are addictive in nature and like a drug addict needing his fix, you standing in the way is going to make him angry with you. It is terribly unfair to you, but that is how it is. The sooner the affair fizzles out the sooner he will wake up and realize what he has done. The 'honeymoon' period cannot last forever.
And like all addictions, they tend to ESCALATE before crashing and burning. A lot of damage can be done as they do, which is why it's best to enforce healthy boundaries.