I know that you are letting this latest situation go, but I have to say that your h is very good at gas lighting. He's thrown down the challenge for people to take photos of the vehicle and the license plate. I think he was asking a few too many quesetions about the time to not have been there. Had he not been there, he would not have asked the time, etc. Then to become defensive and state that you would felt the same way too...gas lighting at its finest! How nice! He came there being nosey to see what he could see and it was not about you being an unfit mother. He wanted to see if you had someone at your home because of the yard man! He'll never tell you that, but I believe your mil is spot on the money for that one!
I'd like to point out something so that you will be aware of it next time. You had a discussion w/him about coming there when you weren't home and in the same discussion you mentioned that you lost your key and would need his. I want to share w/you how he "heard" all of this. Because you had someone come there and sit in the driveway, you now are using an excuse to get your key back because you think it was him, not that you misplaced your key. Then later you find your spare key...see how this looked to him? You aren't the only one that did this...I did it too early on and it was pointed out to me that I should not have requested my key back in the same conversation.
If you can't locate your key, you need to have the locksmith come in and change the locks...you don't know who may have found the key. I would have for someone to break into your home.
Sit quietly, the answers will reveal themselves when you least expect them to. The past is gone, the present is a gift and you need to focus on today, allow the future to reveal itself when it is ready.