DO NOT ENGAGE HIM. He's just trying to push your buttons, and frankly, he doesn't DESERVE your answer.
Just say:
"Just real busy; I'm fine. Have to do _______ , and then had _____ before. Talk to you/see you at (whatever it was you agreed to)."
or
"We're fine, thanks. On my way out to thus-and-such!"
Stay POSITIVE, without being cloyingly so. It'll frustrate him even more.
Now, if when you SEE him, he STILL pushes to know why you don't respond to him more, just say "I've honestly been busy, and haven't had time to think much about you, but if you must force me to give an answer, then I'd tell you that frankly, you don't DESERVE one. I have to go do thus-and-such, I'll talk to you later" (and kiss him on the cheek and leave).
