Let him think what he wants, lol! If most of his friends ignore him and don't talk to him, well, then he won't really notice you are not his friend!