Keep your phone on silent there is no need for you to talk to him late at night or early in the morning if you do not want to. He knows that you don't like it and it amazes me that he still does it. IMHO if he wanted to talk to you and not try to control you he would call at a more respectable hour.
I would also contiue to keep your alone time with him short. If you feel scared of him it is not a good thing to be around him.
Take care and have a good weekend. I hope your degree and the insuing job search pans out for you.
