It would be really easy to see in everything he said that he wants to come back but to start thinking that a will torture you and anyway may not help you to do what you need to.
I hope you don't mind me saying, but you give him way too much access to your time and space.
For starters, he shouldn't get to pop round unless he's invited at a time of your choosing.
Personally I think you need some sort of statement that you can carry in your mind and in your heart that means you don't do any explaining.
Something like.
XH, I loved you throughout our marriage and will always love you, but you divorced me and my life needs to reflect that. I hope you do what you need to to find happiness in whatever path your life takes.
Part of me then says to tell him to take a hike unless/until he is in any kind of state to come back and treat you in the way he should have always treated you.
On the other hand validating him and his concerns, showing warmth and compassion to him does act as quite the lighthouse, but you definitely need to be more mysterious about you.
Hope your otherwise ok.
Both mid 30s, 2 young kids BD 7sep14 XW moved on long ago, now living with OM1 D paperwork in progress