You need to set some boundaries. Why can't he give the key back? Do you have keys to his place, can you come and go there as you wish? NO of course you can't, so he shouldn't be able to do that to you. Put it to him like that. Maybe suggest that you wouldn't want him walking in when you were 'entertaining' remember he has to believe that you've moved on. Good luck
