I suspect the juxtaposition between my being friendly and interested in his day and then setting such clear boundaries is probably confusing the hell out of him. On the one hand I'm interested in the things he has to say, maintaining eye contact and smiling a lot, and on the other I am calmly telling him "this isn't your home".
I even said "enjoy the rest of your evening" when he left tonight (he promptly said he was probably going straight to bed as he needs to be up at 4:30 tomorrow for work). I told him to try and get some chill out time anyway as he's been with the kids for the last 2.5 days.
Not sure where it will go. But I am happy with the path I have decided to take.