OK -- so contact has been re-established. Now YOU be the one to keep control. And you do this by being generally unavailable when he wants you to be and responding to only some of his phone calls and invitations.
I suggest you don't invite him to anything at all for the time being.
