Tell him what you would prefer, regarding the calls, the contact. Make it clear that you don't like it, that it threatens you. Ask him to confirm that he understands that any contact makes you uncomfortable, and makes you feel threatened.
Then let it go.
If he chooses to continue, that's his choice. You can remind him that you would prefer he not be in contact with her, but you can't stop him. You can only choose to cease having contact with him if he continues to disrespect your preferences. Or you can accept that he loves you and will remain loyal to you, but that for reasons of his own individuality, he still will contact her. That's his choice. Yours is to accept that, or not. If you refuse to accept that part of him, to reject that part of him, then you will probably have to reject all of him.
You cannot control him, you have no right to demand that he behave the way you wish. If he behaves other than how you wish, all you can do is change your wish, or leave him alone.
