I understand your fear! I had it, too! And I have still had it, every step of the way. But at each step, I have found that the fear was worse than the fact. I was worried about the kids. And they are fine. (I know yours are younger, but you've also been living separately already.) I was worried about telling family. Went smooth as silk. I was worried that it would bother me as she started to do things to actually leave. No problems at all. I'm actually excited about her house, and it didn't bother me at all when she emptied her stuff from the closet. I guess what I am saying is that I found the fear and dread to be much worse than the actual actions, to this point.
I will never tell anyone to "give up". But I will tell you that I think that stacking your fears if you end it are probably much worse than how things will really be. You've already been raising your kids. I would be shocked if you ended up alone, unless it was your choice. I think there are other fears if you don't end it, too. Look at them, they may be even worse.