You have given him carte blanche to have an affair. And don't minimize it as "only" an EA. Pinhead has a rule of thumb when it comes to marital problems:
If you don't think there's someone else involved, there probably is.
If you think they're only friends, they're probably an EA.
If you think it's just an EA, it's probably progressed to a PA.