I wasn't saying "legally." They are still his children also. He needs to intervene here.
When my wife was having her affair, she was hooking up with her BF during the day, and leaving our two boys (then aged 11 and 14) home along, with nothing to eat in the house sometimes. She would tell them "I'll be home for lunch," and then she'd stay out until 3 or 4 in the afternoon. After this happened the second time, I told her "I can't control what you're doing, but I damned sure aren't going to allow my sons to go without meals. If you're going to be gone more than 3 hours, or if you're going to be gone over a mealtime, I expect you to let me know, so that between me and your parents, we can make arrangements to watch the boys and make sure they're fed."