You know what....

It doesn't matter what you say in the letter, as long as you say no, he is going to think you are a b*tch. Heck he might think you are a b*tch even if you say yes. Don't base your actions on what you expect his reaction to be. Do what is rigt. You don't have to explain it to him. If he wants an explanation, let him ask for it.