I agree with Smiley as well. You don't fit the criteria of a WAW spouse. So get that out of your mind......
fear of him agreeing to stop and hating me <<<< You said this...
BUT, what about the flip of that. What if you, ( heaven forbid )You were to agree to continue or let him continue, then YOU end up hating him, and yourself for allowing yourself to do something you did not want. Flash forward a few years and you then hate him and yourself at the wasted years....... ?? It works both ways.
I have settled in my marriage. Different from this however, but I can tell you with assurance, it's not the way to go! Allowing yourself to do something or settle with something to save the marriage doesn't work. AND, in the end, what are you saving?? You've saved NOTHING.