So I think I still need to go downstairs in the morning and have my coffee down there and then see him off for the day?
Quit basing everything on your H, his moods, his reactions, his will. Go downstairs because you want to. Or stay in your room because you want to. Remove H from the equation!