His argument is that I have off that week and he doesn't. (Though I should add that his job is quite flexible and he can often take time off or "work from home" when he needs to.
He thinks that if I don't have to work because I'm on break it doesn't make sense for him to be responsible for childcare unless I have an actual, out of town vacation planned.
I guess that answers it! Plan an actual out of town vacation!