I have to admit to being a little bit, no a lot, suspicious of the motives a man who manages to "subtly" work into his first converstion with someone that he like older women (well, that leaves you out!) and is totally hairless! While I really understand the ego boost, I don't think I'd call him, and I think if he pushes any more, I would tell him that you really don't think it is a good idea. Because he IS a motivational speaker, he is going to know exactly the things to say to get you where he wants you. You are very vulnerable right now, and my gut is saying he knows that, and is trying to take advantage of it. Obviously I have never met either of you, so this is all based on what you've written, but if I were you, I would be very cautious. And I think that has to mean no coffee, or anything. I really think that if you do anything he will be able to manipulate you, and you won't even know what happened 'til it's over and done.
So, accept it as an ego boost, and a compliment, and run like hell! That's my advice. Remember what you paid for it!