That's a tough one. From his point of view it is going to be really hard for him to see how he did anything wrong. He picked S13 up, spent time with him, and took him to karate. To him, that's going to sound like he was being a parent.
If you don't want him to come in the house, then you'll have to tell him that. But, would you rather have him with S there, or somewhere else? I know it is a mess!
