You know what you're afraid of... you told him. The note was just precious. A creative, proactive big step up from just crying silently and feeling hollow.
I think maybe he is afraid he'll fail to make up for all the others who abandoned, rejected, neglected, and hurt you in the past. I think he has to understand, somehow, that he isn't expected to. In telling him why you are afraid he'll do the same, he wonders if you'll ever trust him with your heart and gets discouraged that you may not ever believe he's on your side.
