WIL,
Of course it's baloney that you are responsible for his confidence.

Lemme ask you this though: As it concerns his confidence, would you consider yourself part of the solution or part of the problem?
Perhaps his idiotic words are nothing more than an immature way of crying for help. I think he has some other problems (enmeshed behavior, depending far too much on others' opinion of him, etc) but there is probably some validity to his complaint, as usually happens with marital disagreements.

Can you try to extract the kernel of truth and ignore the whining, passive way he went about phrasing it?
What is his love language? Do you make attempts to speak it regularly?

If you are confident that you are not part of the problem, then chalk it up to whining and tell him that this convo is finished. And don't allow him to revisit it. Or, rather, don't participate if he attempts to revisit it.
If, however, there are areas in which you tear him down, then for goodness' sakes, STOP!
I read somewhere a while back the following question: Do you use your words to build a person up, or tear them down?

It him me like a ton of bricks. I am a person who likes to joke around but I realized that, somewhere along the way, I had begun to make my H the butt of my jokes instead of laughing *with* him. I was subtly-but-surely tearing him down. Benign tearing him down, sure, it was all in good fun, but the 'build him back up' part was utterly and completely missing. When I realized that he probably viewed me as someone who didn't think much of him, it hit me hard. I stopped then and there and started heaping on the praise. It had an immediate effect on him.

Ymmv, of course.

Take care,
HP