Pup's right, IGNORE what he says.

I would make a small adjustment to your response though.

I woudl NOT say "I'm sorry..."

I would jsut tell him

"Affairs are sleazy, selfish, and cruel, they hurt spouses and children and ruin families -- I will NOT sit back and watch you do this to me OR these children!!!"

If you HAVE to respond, do NOT say I am sorry. And saying people have a right to know isn't going to get the message across that what he's doing is sick...

I was going to add a comment in there about him turning another man into a cuckhold, but I left that out to keep the response simple