Don'tlet this overwhelm you, I know it is VERY HARD. I struggle with my situation daily but have learned how to handle my emotions and myself a whole lot better because of this board, the books, etc...
Try to be cordial, it is not easy when you want to just thwap them with your thumb in the middle of their forehead. Just be distant. "Thanks but I already have other plans" or "Thanks but I am not hungry" kind of thing. That's what I am doing and it garners a more postive reponse than me being curt. Being curt comes much more naturally to me but I have learned that in the long run, being cordial is the best approach.