Actually the other thing different that I have been doing is to ask him questions, like his address which I would have been too scared to do in the past. I am not so scared of his reaction as if he doesn't give me his address it is his loss, not mine.
Maybe talking about small things like this is better than every interaction being about how the other person is etc. Building up a rapport again.