Actually it sounds like you had a good evening. Good company, Good food, and good conversation. Plus who kisses on their first date? Right!!!

And you know what? You did not know if he wanted to go with you? But he picked you up and dropped you off. So next time you see him.
And each time you want to hold him or kiss him. Smile at him. Give him smiles. If he brushes you. Just smile. Enjoy the moment. If he pulls away. Smile. No R talk.

Let him come to you. If he wants to he will. He needs to rebuild trust within himself. And there is nothing you will be able to do to speed that along.

I am glad you had a good evening.

I think your smile got reversed there.