This might go against some of the DB principles, but do you want to subject yourself to what he let's his kids do to you and yours? It doesn't sound like he was willing to change (or even see that he had a problem). But right now, you need to concentrate on you. You need to protect you and your kids.
