I would not be cool with that either. I think you need to get your key back from him. Like i told my H, I don't have a key to his place!