I think you know understand where your at right now and deep down what you have to do but your fear is preventing you from doing it. This is clearly not working for you. Until you are willing to walk away from something that doesn't work for you then you will always be a prisoner in your relationships. We really love to complicate things on this board when it's so simple. What does his actions tell you?
Unlike 99% of the WW husbands on this board, I don't think you H is a horrible person. I don't think he's manipulating you. I just truly believe he thinks your marriage has run its course and is not emotionally intelligent enough to understand limerance and such.
I truly believe in your situation time and space are the only thing that can turn this around long term. If not I think you can expect many years of the same. You have all the power.