One would have to think that by now he would know that you are going to keep the kids safe. Again, if he thinks differently, anything you say isn't going to make any difference!
What you are doing (in his imagination) has him bothered a bit. At this point, I think that's a good thing. He's losing a little bit of control of the picture he has in his head. I like it!
