I don't know if you have anything legal, but maybe a legal separation will set down rules about parenting plan. The younger moms can probably help you with this. I think it's ridiculous he thinks he can do what he wants with your kids and OW without you agreeing.
