So it was about what was convenient for him, really, isn't it?
Why can't you say his behaviour is unacceptable? You can decide - for your own very good reasons - that for the time being you will choose to accept it. But you don't have to accept it. You could get a nanny or arrange some consistent childcare that he would have to fit in with, whether that was convenient to him or not.
