How about a simple "Gee, I wish you had asked me sooner, but I can't I have plans" ?