NO! You do not need anymore than a simple "I don't want to go" or "I want to go"!! Stop thinking about what he is concerned about or what you THINK he wants (yes he says he enjoys your time together, so did my W! It's his ACTIONS that tell a different story!) And stop worrying about whether his offer is "friendly"! he has totally dissed you, treated you as a sex object, lied, talked about other women, the list goes on!
My w's family doesn't think we are getting back together just because I went to a party. It's how you act at that party. Dam it MM, STOP THINKING ABOUT HIM! THINK ABOUT YOU!
The last line in your post says it ALL!! YES HE KNOWS WHAT HE WANTS WITH YOU! HE DOESN'T WANT A R WITH YOU! This is where you are getting hung up, MM. He knows that he doesn't want you the way you want him too! He is absolutely certain of that!