No you are not "loosing" your family. You will ALWAYS have your family. Now the risk is you won't be together as a family. I completely understand this whole like of thinking. My biggest fear is "loosing" my family. It was my wife who basically set me straight in the idea, we are still a family, we will always be a family. It was then I realized yes that is true and my fear is not being together.
So remember they are always your family!
If we really want to love, we must learn how to forgive. - Mother Theresa