Passive aggressive: "Sure, lunch sounds great." Then, not showing up.
Direct: "Thanks for the invitation, that doesn't work for me. Although I care about you as a person, I find right now that I need space from you. Please respect this need."
What is it that you really seek to have happen at the meeting?
