He may have spoken that way because he is depressed.
Next time try to reply with some lighthearted humor. Maybe say, "Yeah, I wish I could say that about myself as well!" Or something much more clever...
Try not to take that stuff personal and any time he talks with anger or says something mean, just let it go. Believe me... there is a small part of them that wants you to "call" them on it and fight. Anything to convince them that they have made the best decision. Don't let yourself fall into it.
I used to imagine myself as Mother Thesera (in fact, I still try to be a little like that... more accepting, more forgiving... more understanding...). But more like a combo Mother Thesera/Marilyn Munroe.
There is no arriving, ever. It is all a continual becoming.