I think that you are right. At some point, you have to know that you have done all you can, but that is ONLY up to you. Dont let anyone else push you (including him) to make a choice that you are not ready to make.
I think that you are also pretty smart to be aware that he might try to use you for your body. Which, I understand that you have had a pretty significant weight loss- Congratulations, by the way! Has he seen you since then? Hopefully he wouldnt do that, but who really knows.
I wouldnt answer his texts/calls immediately, wait at least a few hours. And when you do answer them, keep your responses short, and to the point. Like I said before, tell him that you are just on your way out the door, and cant really talk right now.
I guess I gave the wrong finger to the wrong man...