The only legal thing you could do for not having take the kids and not having him have a word in on abpout it is if you got a restraining order....that is pretty huge and I know you don't want to go down the route but other than that, I would call a police dept and ask them that question....won't hurt anything.
MRHIGHSPEED would know probably on this./...let me see if he is around.