Mooka - H was that desperate to talk to you that he called 4 times and had a tantrum about it, and you respond by being even more distant? What would have happened if you'd responded instead "oh, poor baby, I missed you too!!! Darn that cell phone! Give me a kiss!"
Sounds to me like he's stressed at work and just really needed to see you, and felt rejected when you didn't answer the phone - and now probably feels more rejected.