Well, since no one else will chime in here, I'll add my $.02, for what it's worth.
Your kids are the only part of this that matter to you. Do whatever you have to in order to KNOW they're safe. If you have concerns, bring them up. If they're ignored, use authority, whether in the form of raising your voice, or getting a court order. There's nothing that says you can't get an order of protection keeping someone dangerous from your kids either. Ideally, their father would be all over this. But, it sounds like he's not going to man up.