You're not an idiot, this is your path, use it for the greater good.
My H was always very polite after the BD and said many of the same things your H is saying. He also had an unbreachable wall, occasionally a brick would loosen and he would quickly patch it up. He was a champion at going dark, no personal contact, would only communicate by email (kids), only saw me f2f when he absolutely had to.
Give him the space he needs and more. Your "lost it" moment is a signal that you also need to see less of him. He doesn't need instances that cause him to sit there as you're losing it and think "This is why I'm gone."
Betsey's given you some very good advice. Another book I found helpful was How to Improve Your Relationship Without Talking About It.
Good luck.
Me 57/H 58 M36 S 2.5yrs R 12/13
Let me give up the need to know why things happen as they do. I will never know and constant wondering is constant suffering. Caroline Myss