I am so reluctant to contact this guy's CO. I mean that will blow up into a huge mess.
Look at it this way, QS: which are you willing to sacrifice here, his job...or your marriage?
Because my guess is that if he loses his job or is reprimanded, it will take the sparkle off the affair. He may even end with your wife, he's so ticked. If he goes ballistic and tries to get in your face and get physical, call the cops. Get a restraining order.