In the past my H would ask when I spoke to anyone because he was worried I was saying bad things about him. Now there's a laugh and a half!!
Anyhow, I agree, you have to tell him just for the fact that you are taking the kids but don't invite him. If he does ask to come along. Say that would be great and leave it at that.