Well, if it were me, I probably wouldn't call back, but if I did, it would be later on today and would sound something like..
"Hey its me, got your message. Tonight sounds like fun but I'll have to take a pass, got something else going. Thanks anyway, have a good time, talk to ya soon...Bye..
If you can leave a message along those lines and do it in a happy confident voice, you officially have begun to change how things are going to be from here on out. He obviously expects you to call back or just show up, so here's your opportunity to show him there is a new sherrif in town...