It is really hard to be the LBS. If you were the one who walked away you wouldn't miss him at all. But because you have been rejected you feel very bad. I think a lot of it is in the rejection itself, not who your husband is.
I didn't know how bad it felt until it happened to me. I did not value my H at all. I hated him. And then he got a girlfriend, and suddenly I wanted him back. Then he became the perfect mate for me. Our minds play tricks on us. Just keep looking forward to when your daughter comes home to you. Finally, you will get some time with her.