Guess it comes down to the fact that someone - even our spouse - has as much control over us as we allow them to have.
So, in a sense, he's right. You do have that power.
All you can do is to show him that you won't exercise that power.
How do you do that? Simply by not exercising that power. There's no other way to do it. Of course, it'll take time for him to believe in his heart of hearts that you won't. After all, he kinda thinks that he deserves whatever he gets.
So, the realization that you speak of will only come with time.
But don't make a , Mattie. Like we all say, time is on your side. It's the hell he built around himself, and you don't have to share it. As long as you don't contribute to it.
BTW, I'm not suggesting that you are contributing, all I'm saying is that these feelings are his, and that you can't change that.