Good. now that you've admitted it you know what you have to overcome. Every single person on these boards have fear of one kind or another. The pain that you feel now is that fear leaving the body! Take a deep breath!
No you are not "loosing" your family. You will ALWAYS have your family. Now the risk is you won't be together as a family. I completely understand this whole like of thinking. My biggest fear is "loosing" my family. It was my wife who basically set me straight in the idea, we are still a family, we will always be a family. It was then I realized yes that is true and my fear is not being together.
So remember they are always your family!
If we really want to love, we must learn how to forgive. - Mother Theresa