I think he will see it as justification of his "we're only friends" b.s.

Leave it be. Only apologize for angry outbursts or rude or disrespectful behavior -- never for standing up for fidelity, and for your marriage!

Puppy