I do believe your H is hanging onto quite a bit of anger/resentment towards you (which he's somehow going to have to let go of....IC). What I wanted to share with you though is my own experience recently of not trusting my H.
I've dug through everything....and have honestly never found anything that would lead me to believe he'd ever physically met up with anyone....and now I do believe he never did meet up with anyone. BUTTTTTTTT....I cannot help my thoughts which are a side-effect of his betrayal of my trust in him. I find myself skeptical of his motives for doing things NOW, I find myself second guessing not going places with him (just so I know that's really what he did), I find myself doing a lot of "but what if he did!!!"...these are thoughts I simply cannot always help, although I can control how I deal with and react to them.
There are many people who believe what my H did in our M wasn't all that bad. There are probably many people who would believe that what you did....really wasn't all that bad either, I mean...you two just kissed, you didn't have sex (is their reasoning). The fact is....it WAS that bad to your H.
No matter what you or my H did....it still cuts to the core of a M. I'm still hurt by what my H did (no doubt there)...but I've absolutely had to let go of the anger to move forward. Obviously I'm still going to have triggers that inwardly send me into a panic, but I deal with them. I talk to my H about them when they happen, so he can deal with them as well....I also talk about them to our MC so she can help my H understand.
Fortunately for me, I'm someone who can talk to other people and get things out....heck I'll talk to a lamp post! Your H needs to do this too (not talk to a lamp post, but talk to someone). I really promote him finding an IC...or the two of you finding a very good MC. You two have got to stop playing games with each other....it's like you are in an emotional tug-o-war, and that's not productive.