Ewww...I don't like the turn of these conversations. Do you think he's possibly preparing himself for exiting the marriage?
Why would he be afraid you would hurt your daughter? I mean he's the one who gets violent by pushing, pursuing & yelling...unless you've left something very important out, which I don't think you have.
You have EVERY right when he is pursuing and yelling to turn around and tell him that, "I'm not hearing you when you yell at me, stop yelling and talk to me like an adult and I'll be happy to listen."