I agree with everyone else - do what you think is right for you.

It is obvious that he wants to talk to you. If you don't want that then tell him.

Me late 50's
M 9/06
D 4/11