If you can't legally change the locks, then add another lock to the doors and don't give him a key. You will be truthful when you say that you didn't change the current locks and do not give him a key to the ones. If asked about it, say, after having a convicted felon in my home, I do not want him to return and come into my home and steal stuff either while I am away or even here. I do not know the company you've been keeping, therefore, I do not know who has a key to our home and who doesn't.
There is always a way around these things and you have to think outside the box.
Sit quietly, the answers will reveal themselves when you least expect them to. The past is gone, the present is a gift and you need to focus on today, allow the future to reveal itself when it is ready.
