Maybe H is asking "why" because he is getting nervous that you are getting on with your life. What he really wants to hear (for his own reassurance) is "because I had nothing better to do".
Of course, you wouldn't answer this way because it would not be true
Sounds like your H is getting jumpy because he sees you have a good life without him there.
