I think it's great he is communicating his discomfort to you. Much better than wordlessly throwing up a wall and freezing you out in my opinion.
I'd suggest that anything you can do to communicate to him with your energy, so to speak, that you are not actively resisting his decisions, will help him feel more comfortable around you. I know it can be hard to find the middle ground between coming off as hurt and withdrawn on the one hand and carefree on the other. I sometimes find sober, attentive and compassionate to be about right for me.
