I would change the locks on him, no need to ask for the key back. You must not make his decision any more comfortable to him. Letting him use your stuff anytime he wants is not a good thing. Having to ask you to use it is much better don't you think?
Great job on the galing. Just know that you should not contact him, he will contact you. When he gets mad about the keys not working, just say, I'm sorry, but I did not feel safe being here alone, and thought changing the locks would be safer for me.
The open door policy that you have is not letting him feel that he is going to lose you, and that is when turnarounds happen.
