I just don't see how he's comparing a prior R from before we were even dating to having an EA WHILE we are married. We're talking apples and oranges here, but he's just so stubborn.
If you study affairs at all, this is actually pretty "SCRIPT." It's a projection thing -- THEY are doing something that they know isn't right, so then they do this moral-equivalency thing where they accuse YOU of it. I've seen it either where they KNOW the comparison is ridiculous, or where they're so paranoid from their own immoral behavior that they actually do think you're up to something.
You handled it well -- unfriend the guy, to call your husband's bluff. But you should have gotten him to agree to the same.