I think I'd posted earlier this week and asked why you always stayed around when he comes to see the children. I would let him have them by himself. After all, if it does come to D, that's how it will be for him -- at least for the most part. How convenient for him that you look after the children all week, and then when he decides to grace you all w/is presence on the weekends -- you hang around.
I would go out and stay out and be vague about your plans. If he doesn't like it - so what? Hanging around him all day in the hopes that he'll talk about himself a little seems clingy. If you can, try this 180 and see.
You are doing great, and in the end, only you know what's right for you.