You did the right thing. No big deal and move on. Nothing changes. You're not interested in being friends. Anything short of that " sorry gotta run".