So he has her whenever he wants and he has access to his family whenever he wants. Sounds like a perfect piece of cake for him. He's loving life. He doesn't want the family. He has walked out on the family yet still wants the family available to him when he is gracious enough to allow you and the kids to spend some time in the presence of his greatness. What an utterly soul crushing existence for you. My heart breaks for your situation but angers for it too because I know it could be different.
This man chose some skank over his family. He has to learn what that means. He can have the skank. What he can't have is access to you and the kiddos. Not anymore. That's not how life works. Make him feel that pain. He deserves it and it could wise his dumb a$$ up. He has to feel real pain and loss or there is zero reason for him to change.