Yeah, I see your point. I don't think you haven't forgiven yourself. I also don't see you wallowing in guilt. I think that SHE hasn't forgiven you... and given what you've said about her here, probably never will.
In that sense, while you may not be feeling guilty, and you may feel morally justified and right about your decision to come back. you are allowing her to punish you day after day. So even if you're not guilty, you're still doing time for your crime. This is a life sentence.
I think she is the one who cannot put the D behind her. The reason I think she regards the D as a bigger crime than the A is that an A is mostly a private offense-- a major offense, but private. Even if other people know about it, it's not official, it's something that can be kept under wraps, if the parties involved so desire. And when you got into the A, you did the right thing, you left so you could be with this other woman. So your W can't fault you for cheating behind her back for years.
But the D is a public, legal declaration that you no longer want to be married to this person. That you have reached a point where you want to take back the promises that you made in front of friends and family. It's taking something that you two created and holding its head under water until it stops breathing. It is a cruel and destructive act-- often necessary, but still, a public execution.
I would guess the D was much more humiliating to her than the A, because regarding the A, YOU were WRONG, so your W could stand loftily on the moral high ground. "He did me wrong, he's an @sshole, he left me for an OW."
By leaving your W and divorcing her, you publicly and legally straightened out the situation. That was the right thing to do at the time, but a public humiliation to the person who was left.
For your own reasons, you further decided that the right thing to do was to come back to the marriage. She took you back, so she's not in a moral position to fault you for that... but she can sure the h*ll punish your @ss for the rest of your life for what you put her through.
You've paid the price, you've done the time, you've made amends-- WHEN is she going to let you out of jail?