Dia!
You seem to be on the right path...
You knew there was going to be a pullback, so it didn't catch you off guard, of change your attitude. Keep things comfortable for him, the next time he may move a little closer. Your patience is going to be truly tested.