Mine does this too. And some of the things that he brings up as being so damaging to his self worth seem actually really silly. (like my reacting to his aggressive driving when he was practicing was somehow so damaging he never wanted to try driving again).
I think it comes from a place of insecurity, but also I think it goes back to childhood issues of feeling unwanted or unworthy... like they were looking for every excuse or opportunity to say "Aha! I knew it! You rejected me too! You hurt me too! You didn't think I was good enough either!" I think a lot of it is projection of other bigger, deeper seeded problems onto molehills.
I think they also use a lot of these things to mask their own fears and feelings on inadequacy... so in my H's case he felt somewhat frightened of driving because he knows he is something of an adrenaline junkie and loses his temper easily, that driving could be a source of problems and big expenses for us if he had an accident. But rather than admitting that and taking personal responsibility to dial back his behavior, be more cautious, address the anger management, etc ... it was easier to put the blame on ME than on himself.
me-35 WAS-37 T-16 1/2 Son-14 (HF Aspergers) BD,ILYBINILWY,"I met my soulmate": Oct5,2013 "Letting go because I love him, holding on because I love him."