Also.. don't get mad or offended. Just simply say, "oh, I already have plans. I'll be home at xxx time." If he asks, say, "I'm meeting some people I don't think you know. Not exactly sure what were are doing yet." But don't offer if he doesn't ask.
(And take him at his word when he says "no big deal".)
