Don't reply. Next time he says those things to you in person, you only have one or two responses:
"Everything I've done, I've done to fight for our marriage and our family."
"I decided that I was no longer willing to lie to cover up your affair."
If he says something about contacting YOUR family, calmly say "Knock yourself out -- I have no secrets from them. I'm sure they'd love to hear from you about your behavior."
