Is it possible that H is using this as a way of getting back at you? Viewing this as an affair of his own, but since you are there, it's really not as bad as what you did?

I've read that affairs are the triangulation of relationships. Three people involved. It's what soap operas are made of.

I'm a simple guy, two's company, three's a crowd. I wouldn't do it.