Good grief! Well again - you are NOT trying to steal his family away from him you are merely maintaining some rather long standing friendships with people he happens to be related to. His getting kicked out of the house had NOTHING to do with you maintaining contact with his family. He sounds very paranoid to me.
It seems to me that he thinks that as long as you are involved with his family you will remain the good guy and he is the bad guy. If he can just get you the "good guy" out of the picture then the family will begin to like him again because on the surface it will look like you abandoned them and he hung in there.
Not sure you have to deal with this though. You can simply choose to ignore it and be friends with whoever your choose.
