Ugh Red, I definitely think you're on the right path with getting the distance and setting boundaries. The thing about friendship is you just don't know what the future holds there yet. Of course he wants to be besties, HELLOOOO! Look at your fine self, how you've handled all of this. That is a compliment!
I'm in the same situation, as my WAH is a good man despite the alien invasion. Even when hurtful things come out of his mouth, they hurt because that's how I receive it. To him, it's objective. It's like they're not operating in a place of feelings because it's their decision. If it were ours, I'm sure we'd be the same.
I say stick to your boundaries, but don't shut out the idea of a friendship yet. What if in the future you do become great friends, and do share some aspect of your lives together. With children, you always will have some sort of relationship. Life is too short for it to be a bitter, callous one. Too short.