I don't see why you have to lose his family? Your relationship with them has nothing to do with him.