I might just answer with a "No bother. What's up?". There's no sense mind reading why he starts out the conversation that way, but I think by answering in a quick, no-nonsense manner you may be able to get on with the conversation without any awkward pauses.
In case you aren't already doing this, I would also suggest letting his calls go to voice mail more often than not and then returning his calls later that morning, during lunch hour, etc. Let him wonder why you aren't jumping every time he rings.
