Just wanted to add to the C's comment about the lack of QT together. IMO, time spent with you is something H should have to work for. It shouldn't just be handed to him on a silver platter. Look where it's gotten the two of you so far.
If you're not going to go dark, then you should think about accepting only some invitations. Not all. Make him work for it.
Getting over a painful experience is much like crossing monkey bars. You have to let go at some point in order to move forward. ~ Joseph Campbell