Don't beat yourself up...you are in such a painful situation! One of my books (Torn Asunder) says not to worry about getting angry at your cheating spouse and pushing them into the arms of the OW--THEY ARE ALREADY WITH THE OW! You haven't made anything worse. If anything, they need to see your pain for them to ever hit the bottom and try to improve.
Your H sounds like mine when I have gotten angry or pushed his buttons about the A. They lash out to justify what they've done. I'm also not convinced that in their wacky minds...they even remember much of any conversations (good or bad)
Hang in there and take care of yourself. YOU don't deserve this!!!
M 44, H 46 D11, D9, D5 Married 12 years PA confirmed 9/2011 I filed 3/2012 H moved out 7/2012