Yes, in my opinion, there are times when you should leave a marriage. I would say that when a man abuses his wife or his kids, then the best option is to leave a marriage.
It is too bad that you live in the same town as OM and will run into him. I guess I should remind you that talking to him, even in a public place, breaks the no contact rule. Now you are a mess. Do you have an Individual Counselor? You need to talk to someone about your lack of sexual desire for your husband. You need to work through it and make your marriage a real marriage. Not a friendship relationship. I really think you miss the sexual relationship you had with OM. There's no reason you can't have one with your husband. But the fact is that you don't. And I don't believe the problem is all your husband's. You need to work through your problem. It is not going away by itself.