Actually it's a good thing when he inquires about why you didn't return HIS call.
It is a positive. They get so use to knowing exactly where we are and what we're doing that any CHANGE from that norm throws them for a loop....and YES it does make them wonder and a little nervous when all of a sudden we are no longer "staying put."