I was always the first to hang up and made it evident that we had lots to do. So, oddly, he seemed to react more positively... Is there an underlying reason, i.e. that he doesn't want too much intimacy or he needs his space?
Don't know if they're applicable to your sitch, but maybe these posts may help shed some light?