He said yes to dinner---he has the ability to say no. Rejoice in that, he chose to be with you that evening.
Hesitation or not, he seems to want to be with you, I'd keep asking him out at appropriate intervals, eventually he'll start expecting them and maybe the intervals will be too long for him.
Wish I had suggestions on not going crazy, but thought KAW had some.
