How would you treat someone else who stayed and watched the cats and took care of the house for you? I think the most important part of a relationship is the communication. So I think asking a favor of him, and then dismissing him afterwards is cold and heartless. I think you should have a heartfelt conversation with him if he brings it up. And you should make an effort to be considerate of his feelings in the things you say.