"What should I do for my H?" ASK HIM. Don't accept a muttered, "I don't know" or "I shouldn't have to tell you." Make a list of what he tells you, and ask him to be specific--"Go to museum once a month" is better than "Do more things together."
Of course, he'll have to be in the mood to really communicate and not consumed by resentment from your affair. Ask him if he is open to discussing this in a meaningful way, and assure him that your commitment to meeting his needs is real. Tell him you know you didn't do the best job in the past and that you want to change [if this is true, Jane].
Then you tell him, gently, if he is receptive, what you need--including attention to his physical self (be kind: "You look so good when you...")
Then, dammit, the two of you need to start doing that. It's not hard. And you don't have to agree with his needs and they don't have to be like yours. You just need to meet them, in a way that you find agreeable (and sometimes you simply give out of the goodness of your heart). The same is true for him--he should do things that please you whether he thinks he "should" or not.
"Why can't I be happy?" I suspect the clues are in your troubled past. I also think you are spiritually directionless. You need competent counseling and you need to ask God to move in your life.
"Why am I the one trying?" Because that's what mature love does. Mature love keeps reaching out, even when the other parnter doesn't seem to be doing a damn thing. Love is patient, love is kind, love doesn't keep a list of wrongs, says the New Testament.
"Why do my actions have to impact so many people?" Because you live in the world. Because you chose to marry and therefore you have a certain responsibility to try to maintain the happiness of that person (you are not SOLELY responsible for his happiness, nor he for yours, but you do have to actively contribute). Your actions matter. It's just that simple. If you act in ways that hurt others, you have to face the consequences.
That said, you can't do much for others until you are healed inside yourself. That's why I think you have a lot of internal work to do. And while you get yourself straightened out, try to do for your H as best you can--you married him, you owe it to him.
Do you all have a child together? Does your husband still want you, no matter what has happened? If either is true, you better leave no stone unturned before you decide to throw in the towel. And if you do throw in the towel on either the father of a your child or a man who loves you still, you will have to accept the consequences. They are far-reaching and they are not small.
But if you two have no children and your H is not much committted to the relationship, maybe you should both be honest with yourselves. Biblically, marriage is a convenant not to be dissolved--to do so for any reason other than abandonment or infidelity is a big no-no in that world view. But you have committed infidelity, which might just mean you aren't going to stick with your H for life no matter what.
That's the question you and only you can answer. And yes, you are faced with the burden of choice and with the consequences of the choice you make. No two ways about it.