I just gotta say that I agree with everyone. And, you do to! You've said as much.
It don't mean nothin'
Treat it that way.
I have a strong suspicion that the "fantasy that never happened" was resolved a long time ago - at least as far as your H is concerned.
So, you have to resolve it in your own head.
It's a tall order. You know that I have my own issues, and that I'm still dealing with 'em. I can't honestly say that I'll deal with them any time soon, but the best thing for me is to keep W out of it.
It's kinda like you always say. Don't let him/her suck you into their drama. As hard as it is, you hafta act in a way that shows you're not trying to suck him into yours.
Because rightly or wrongly, this is your drama as far as your H is concerned.