I don't think your solution would work for me. You see, I have been arguing with my wife, that I want way more than sex. I want closeness, intimacy, I want to be in her soul, not just her vagina.
If I argue that she should not care about me having an affair because she doesn't care about sex, I would be undermining my own position over the meaning of sex to me. I would be justifying her suspicion that all I am doing is looking to get my rocks off.
She and I both know that if had an affair, it would probably not be a one night stand, and that I would be playing for keeps. Unlike other men I know, I have never been capable of casual sex.
Of course, I cannot speak for you. You seem to get the most milage out of your H by acting like the dissatisfied customer, and he is the eager supplier: you are Ford and he is Visteon. Your attitude is that you like his product, but if he can't supply on time, you will just shop the business around. After all, parts are a commodity. It's unfortunate for the both of you that he can't just give for the joy of giving.
SM
"If we will be quiet and ready enough, we shall find compensation in every disappointment." Henry David Thoreau