Sorry to hear about your snitch, I don't have a child either and that is what through my H into his MLC/Depression. However my H has 2 D's from his first M, that I am now worried wont be in my life, its a terrible feeling of loss.
You are a strong person, going on fertility drugs and all that involves then still think of walking away so he can have a family. I don't know how long you where trying but I have a cousin that tryed different things for three years, it didn't happen so they adopted a baby, 11 months later they where pregrant. Funny how God sometimes plans these things.
I know how you feel about being old, i'm 37, 38 in August, I only have a year maybe two myself, which also infuriates me because he knew full well I wanted to have a baby when I was ready and he said ok more than once (sorry always get a little upset when I thing about it)
It is very possible that your H will decide trying to raise another person children is not easy and get out. While I have only had the girls on weekends (met H when one was 6mts the other4yrs), there have been 100's of times they would just sit in front of the tv without even looking at me when I tryed talking to them. Raising childern is a hard enough job, but when they aren't yours its much harder. However in my case it turned out great they are both teens now and we have fun together.
Keep strong, I don't know how old you are but its never to late to adopt on your own!