You could say something like "I'm really busy tomorrow, how bout the day after." That way it's on your terms, not his. And he doesn't get the reward for being able to chose when he calls.
M:26 H:32 T:8yrs ILYBNILWY/ "I'm not happy": 2/6/16 D-Day-discovered PA/EA: 2/10/16 Separated: 2/14/16 He's living with OW I moved to different state: 06/16 Currently: Trying to move forward