Yeah, stick to the high road. If he brings it up again, you can say simply: "I do not disparage you to the children and I expect you to do the same".