Hey M,
Wow, I am surprised, it sounds like a really cosy, romantic, stress free setting.. and he WANTED to go there, with YOU, so I dont get it. Maybe its just that he feels the same way you do, you are both cautious/scared/non-plussed, whatever and the effect is magnified if you're both that way?? But then, he left, he wanted back, he's back, so the onus should be on him to instigate the romance, IMHO.
Could you ask him why he didnt ??? And why he left at midnight??

Happy New Year to you.. I missed you too!!
Al xxx