This is a great topic to go towards--second marriages or more. I had all of that stuff thrown in. I would be lying if I claimed that blending a family is easy. It isn't. And everyone, even the kids, carry baggage with them.
But you gave a laundry list of reasons why you can't start over. So you are single for life? Or are you going to wait on Andy? What if he marries again? I think that the difficulties associated with second marriages are real, especially when kids are involved, but it shouldn't completely deter you from trying again. All of these issues would just have to come out prior to the marriage so there are no surprises.
During the times you were "dark" from Andy was he nasty then? When you don't have contact does he get nasty? I'm not saying move away physically, I was talking about emotionally. He'll get the hint that you've quit. Him getting nasty about it isn't a reasonable excuse to remain in limbo until you die. You are cheating yourself and I still can't comprehend why. My first XW didn't take my relationship with my second wife that well, but I just avoided her more. It's pretty hard to be nasty when you don't talk or see each other. It's not easy, but you owe it to yourself not to let anyone hold you back from happiness. If Andy wanted you he would have made more effort.
In the end, it's not the years in your life that count. It's the life in your years. Abraham Lincoln
It is hard to fail, but it is worse never to have tried to succeed. Theodore Roosevelt
