You don't necessarily have to play hard to get. When was the last time you spoke with him? If it's been a while, just go over and say hi, ask him how he is and then leave. Don't start any conversations with him, let him lead.
Just say you don't want the D, be calm, don't let him push buttons or make you say something you don't want to say either.
