I don't understand isn't there a schedule? Why is he even asking? As much as we all hate him here the kids do need to be with him. Try to think of it as their time with dad and your time to yourself instead of giving things up for him.
If they don't know when they're going, they might be wondering.
Resentment occurs when we aren't doing what we need to care for ourselves, though we expect others to do it for us.