You have said the trust is broken. My W didn't tan for 8 months while we where separated, but the second she had a babysitter(me) or we got back together she started tanning everynight. I gave her trust, but in the back of my mind I didn't. Gone for 1-2 hours a night tanning. WTF?
I think you better get that trust back. If that is quiting or compromise, it doesn't matter. Just get the trust back. If you regain trust I'm sure he won't care how much you go to karate.