I wouldn't think anything of it. If it really bothers you, ask - matter of factly with no emotion or guilt or anything. Just "why didn't you call me directly?"
Otherwise, just let it drop. It's one of those silly things that would bug me but don't really make a difference.
Have you told your XH that you want him to come by to visit when he knows you are home? Maybe he has his own hangups and thinks you don't want him around.
