You could try asking in a different way that doesn't make him defensive or shame him. He knows that it makes you feel isolated to call his dad and it is his reaction to you.
Change how you act towards him and he will have no alternative but to change his REaction to you.
