You've already gotten a ton of good advice. I would echo what Ellie and Lee wrote about love languages (I still need to read that book!) and ways to feel closer. I sense that both you and your H really take each others' moods personally, and that gets you two "stuck." Detachment from the other's mood would be a good thing! If he's in a bad mood, ask him if there's anything you can do to help or if he wants to talk, etc., and then go do something by yourself until he cheers up! Or do something fun together and banish those OW thoughts from your mind.
Believe me, I know how hard it is. I'm frequently wondering what H is thinking when we're together, but I picture that stop sign in my head and try to enjoy just being with him.
