I would ask for the key back. If he balks, ask for a key to his flat. If he refuses, tell him that either he can give the key back, or you can have the locks replaced again.
No more Mrs. Nice Girl!
M(53), W(54),D(19) M-23, T-25 Bomb Drop - Dec.23, 2017 Ring and Piecing since March 2018