His time is coming.. maybe not when you expect it... but it coming.
If he ever says anything like that again.. Look him dead in the eye.. Smile.. and say "I am not the one that left." Or if you just wanted to be mean.. do all the same thinks.. and say "Sucks to be you."
Relax Eat Think Act normal React.. Smartly. Do something different. Emulate. Do Work.