It's really really difficult to get this right. I know I wasn't at all sympathetic and understanding about my H's EA (which is nowhere near as serious or long-lasting or deep seated as your H's behaviour, May - I do acknowledge that). Instead, I punished him like crazy and used his guilt to get the closeness and affection I wanted to the point where the man must have felt like some kind of vending machine - every time I felt bad I went to him and demanded a dose of contrition and comforting.
There's a middle way and I had to find it from the other direction. But it did involve refusing to buy his stories about what he did and why he did it and whose responsibility it was, and refusing to dump the work of recovering onto him.