I think that's a pretty good response.

I also like "I'm not ignoring you; I just have some things I need to take care of." (implies YOU are moving on, and let's him twist in the wind as to what those "things" are!)

or

"This is your mess; it's not my job to rescue you from it."

or -- my personal favorite:

"I don't want to."

Puppy