This really is bullying, and given your situation he really should be ashamed of himself.
He won't do this in public right? He won't do this in front of anyone else while there are witnesses?
If not, then it is bullying and it should be regarded as such. Handle as you are now... avoid an instance of being alone with him. Ignore all phone calls, reply in writing - email to everything... be sure to cc a support team member so he can see you aren't going to hide his bullying either.
