He is emotionally manipulating you by threatening to call the police, etc. Why does it scare you? Do you think the cops will come take your children out of their home, based on just the word of your H? I mean, what is he going to say if he calls them? "My W won't let me see my kids"? They will tell him to get a lawyer or take it to court. And you have enough documentation for a Judge to see who the irresponsible parent is.
Are the kids sleeping at the same house your H is staying?
It is not about what you feel should work in your M. It is about doing the work that gets the right results. Do what works!