You aren't D or "D", you are separated. It is very clear that neither of you is emotionally D. That isn't a bad thing. Nor, do either of you have to be emotionally D to detach to a healthy degree. Unfortunately, many people won't let go which forces emotional D as the only solution.
Anyway, why can't he take the kids during the time he spends at your house? Why can't he take them weekends, or one day over a weekend? Have you tried to work with him to find a schedule that works for both of you or are you taking his visitation process as your sole responsibility?
True. This is what I was thinking when I asked him to move out. So, what am I doing wrong? How come I feel like no matter what I do or don't do; whatever I say or don't say just isn't right?
Because "right" in your book means "turns SO around, slaps some sense into him, gets him back home on his knees admitting his horrible mistakes and declaring his undying love." Unfortunately neither your choice of actions nor your choice of words has the power to do that. So, by that understanding of "right" your actions and words will always be "wrong."
How about understanding "right" as "respects me and others as well as I can given my understanding at the time and emotional resources available to me"? On that understanding, I'd say you've been doing A LOT RIGHT lately.