My .02 is that he just can't stand the thought that you're not there where he left you and it's bugging the crap out of him.

I could be wrong, heaven knows that's happened before