First off, ignore everything your H is saying right now. As DB/DR says, believe nothing of what they say and only half of what they do. Your history with him is real, but his pain and confusion right now has put blinders on him looking at that history. He needs something to blame for the pain, and he can't blame himself so he has to blame you and your R. Believe in yourself and your feelings.
"I made the wall of shadow draw back, beyond desire and act, I walked on.
Oh flesh, my own flesh, woman whom I loved and lost, I summon you in the moist hour, I raise my song to you."