Hey Limbo, I just wanted to jump in here and recommend a book to you. I don't know if you have read it but I just got "Not Just Friends" by Shirley Glass a couple of weeks ago. I have a lot of the same questions as you and it has really helped me work through my feelings.
I also think my H didn't understand the depth of the affect his A had on me. He unintentionaly (I think) made me feel that my feelings were irrational. It was amazing how much that book validated my feelings. It also give you a plan on how to work through those feelings.
I think if what happened is swept under the rug then it will probably happen again...and it will always feel like the elephant in the room. I'm glad he is willing to sit down and talk to you about it, that says a lot.