Maybe you need to set some boundaries. Tell him you like spending time with him and the kids but that the verbal and now physical abuse will not stand. Point out that you are protecting yourself but also that you're protecting your kids from having to see this kind of behavior. Tell him if it happens that way again, he will still be able to see the kids but not with you, not in your house (he moved out, that makes it yours), and not with the same level of flexibility. He's enjoying a lot of benefits right now in terms of access to you and your kids. I'm not suggesting that you threaten him, just that you lay out where the boundaries are and what will happen if he crosses them again. Be respectful but firm. Something like I know you're under a lot of pressure but I can't allow myself or my kids to be part of your reactions to that pressure any longer. It may push him away but for my $.02, I not sure that's 100% a bad thing right now.
