I have some abandonment issues as well that began when I was around 10 years old, it still affects me to this day. I wish I could provide some advice that could help you but I'm kind of in the same boat that you are.
Hang on your hat on the good things, most notably your relationship with D. You have been a rock for her and that speaks to your character.
You're a good dude, Rooskers. Don't let anyone tell you otherwise.