Yes, I would back off, continue doing things that will build trust, afterall, there is a lot of it to do. You may feel used, but he is looking for you to either want to do things for him, or to go back to they way you use to be. do not over persue him, but make yourself available when needed. A visit to his place once in a while is fine, but no pressure.
