Sounds like he is a controlling ass. Do not accept it, and I would be damn sure I would not care whether or not he would be mad at me for ignoring him. He is the one that chose to leave, he has a phone, he has a car to drop by, why should he be the one to be mad at you for no contact.
If you were a man I would say to get some balls, but, well, I guess I have to say it is time to be a strong woman for a change. When he does contact you and he will, make sure you do not stay on that phone long, you are much too busy. Do not ask, how he has been doing, or why don't you come home or sorry I have not called. That is what he wants you to do to get back in control of the situation.
I guarantee you he is wondering why the hell you have not called, and that is a good thing BELIEVE ME.
YOU ARE MUCH TOO SMART, PRETTY AND FUN to be treated like this, when your attitude is he does not deserve me is when your situation will change. Until then, I am affraid it is doomed.
He will call, get a plan on what to say and how to get off the phone first, remember be cool, pleasant and busy, speak to him like a friend of a friend, you know of him but not intimently.