Has he told you what taking responsibility for your extra marital Rs would look like to him? If yes, is it reasonable? If no, ask him what else you are supposed to do.
I was going to touch on this as well. There is a book out there much like 5LL called Five Languages of Apologies. The gist is the same....different people forgive and seek apologies in different formats. We took a test in retrouvaille to find out which we had and most people gravitate towards two. They are:
Apology Language #1 -Expressing Regret: “I am sorry.” List the hurtful effects of your action. Show remorse. It doesn’t count if the person is only sorry that they got caught!
Apology Language #2 – Accepting Responsibility: “I was wrong.” Name your mistake and accept fault. Note that it is easier to say “You are right” than “I am wrong”, but the latter carries more weight.
Apology Language #3 – Restitution- Making Amends: “What can I do to make it right?” How are they now? Is any debt owed or repayment due? How shall I make amends to you? Do they need help dusting themselves off and getting back up on their feet?
Apology Language #4 – Repentance: “I’ll try not to do that again.” Repentance- literally means turning around 180 degrees. Engage in problem-solving. Don’t make excuses.
Apology Language #5 - Requesting Forgiveness: “Will you please forgive me?” Be patient in seeking forgiveness and reconciliation. They may need some time or greater clarification of your input from Apology Languages 1 – 4.
My W had so much resentment built up regarding my past behavior, it seemed like saying sorry wasn't making a dent and I refused to say sorry over and over, so I kinda came up with a different idea....being extremely specific about things I was sorry about and folding some of these different languages into that. She said it did more to improve our relationship over the course of a few months than anything I had done in the 14 years prior. She still hasn't forgiven me completely, but that's on her. I've owned it, I've told her how I feel about it, how sorry I am, how much I appreciate the things she's done, how I've changed so that it doesn't happen again, etc.
If you choose to do something like this, I think the mindset is extremely important. It's not about making things better (though hopefully it will). It's really about these languages...accepting responsibility for your actions, expressing regret for hurting your H's feelings, repenting - working on you so that you don't repeat the same behavior, etc.
It's up to your H what he does with it, and he may be so bent on punishing you that it doesn't matter to him. But what's important here is that once you've done these things, you can stop accepting that punishment from him as you've been doing.