I have great compassion for you and also for your H. My bf was raised by a very sexually inappropriate single mom, too, and was exposed to stuff in childhood that left him wary of women and wary of sex. I only found out about a month ago that his mom used to come in the bathroom when he was showering (at about age 8) and fondle him under the pretext of making sure he had washed thoroughly. When I heard that I burst into tears for the little boy who was naked, vulnerable and trapped. Where was he to go? What was he to do? All he could do was put up a wall to keep her and other women out. My bf's mom owns a bar and is STILL sexually inappropriate at age 89! Recently before we went out with my sister-in-law for T-giving dinner my bf had to sit his mom down and tell her not to use sexually inappropriate language at the table (e.g., no "pussy," "tits," etc.) It was the kind of lecture you give to your 17-year old, not your mother! Yes, this lecture was necessary. At my first T-giving with the two of them four years ago, I met her for the first time, but that did not stop her from telling us how every guy who comes into the bar notices how big her boobs are!
It was perfectly natural for you to blame yourself all of these years-- I too have fallen for the "all men want sex all the time" myth and assumed there was something wrong with me. Please do not call yourself a failure! You are on a path- you have gained insight about yourself and your H. This is not failure.
So. You've figured out that there's nothing wrong with you. Good.
There's also nothing WRONG with your H either. He adapted to what happened to him in the only way at his disposal back then. Unfortunately that adaptation is blocking any intimacy between you and him. Very likely in his other relationships with women he also unconsciously chose women who would treat him like sh!t in order to reinforce his unconscious beliefs about women.
But you're different. You aren't like the others.
You know that underneath the obnoxious man is a frightened, wounded boy doing the best he can to protect himself. You may be the first woman in his life to know that.
You still don't have to stay in this R. You can divorce him. I don't know how old you are, but it doesn't matter. You are entitled to happiness at any age.
You can also approach your marriage from the perspective of these new insights. You can look at him through these new eyes and forgive him over and over again. Also forgive yourself over and over again-- for putting up with it, for making it your fault, for being angry at him, for making the way you feel his fault.
There is work to do whether you stay or whether you go.
An affair is not the answer.
A new relationship might be the answer, if you first leave this one. Or possibly a separation so you two can get some distance and perspective between you.