You said you can't live like this, so do something about it. Leave if you must, but holding an affair over his head (I will do this if you don't step up to the plate) is disgusting.
How would you feel if the shoe was on the other foot? That's really a horrible thing to say/do to another person.
Leave him! He doesn't want what you want and you are not willing to work WITH him. Both of you are miserable with the other and an affair will only make things worse.
For your next letter, why don't you write something like: Either we go to marriage counseling and work through our problems, or the M is over.
