Tell him you already made plans for nite. Tell him you might we able to do Friday lunch, or maybe something on Sunday.
He's going to do what he's going to do. There's nothing that says you have to make it easier for him big you think you can handle it, let him sweat a little.