Was he always so remote, Dilly? What was he like when he was the pursuer?
From some of the things that you say about him - like him wanting to end the interactions first, or at least control when they will end by telling you when he needs to leave - it almost makes me wonder if he is DBing you.
Have you ever laid your cards on the table? Given him remorse and an outline of what will change? Do you think he is waiting to see something in particular from you before he will open up again? I know you don't have R-talks and that this improves the general interactions from you. It's also clear he wants your attention and doesn't like it when he doesn't have it - and that he's willing to initiate dates etc. AND that he's very guarded. Do you think there's something in particular he is after, or is this just a case of needing more time?
How are his relationships with the kids and how does he talk about what he is doing and why he is doing it with them?