Wooba, your H's fixation on you seeing someone else is his projection of his own affair. I didn't catch it first time through, but it's pretty common on most threads here. So no need to sit there and ask "Am I crazy? Did I say something once upon a time about that?". Likely you didn't.
Gaslighting at it's finest. I'm not sure that H (or anyone in an A) is actively TRYING to make their LBS feel crazy. I don't think it's the active plan in their mind. But they do. By projecting their own guilt and accusing US of being the unfaithful ones.
My personal (non) favorite is the telling us that we will be better off without them. I mean, we will of course be fine. I am fine. My life is quite full and wonderful. But it was invaliding of my feelings to be told by XW that I would be better without her, or that "this is for the best". I realize now part of what makes those sentences hurt so much isn't about being left, but about not being heard by the person who claims to know us best and have our back.