Don't help him out, leave his ass alone, this is his bed he is making and it sounds like you want it to make it comfortable for him.
You are correct on the backslides, this day was a perfect opportunity not to be there but you were all over him, with lunch, do you love me, get our picture back up. How long was he there, and all of this happened from you.
OK now it is over, you MUST react cognitively and not emotionally. Think before you say anything, and in fact say nothing unless spoken to.
I believe it is time to go very dark, but I do not think you have it in you and it would be the very best way to get him to be curious about you.
Do not ask for the keys unless he starts to use them and you are not comfortable with that. Then just ask him to leave them under the mat the next time he is there on his way out. Or if he really pisses you off and is coming over all of the time, then change the locks.
TIME TO GET STARTED, You have not yet begun. YOU CAN DO THIS!
Let's see how long it takes for him to contact you without you contacting him. Burt