Regardless of what direction you take, you don't want to stay where you are at. You are miserable. You need to find the courage to make some changes.
Keep in mind, the 2 options that you describe (1. end the marriage and commit to your affair partner, 2. accept your situation and stay miserable) are nowhere near the only options that you have. There are many things you can do between these extremes.
Where do you see yourself in 5 years? Where do you want to be? how do you get there?
FYI - There is so much wrong with your husbands thinking. It is also wrong to turn outside the marriage to fix problems within it. Again, something needs to change.
