Haven't followed your sitch but That is NOT ok. Protect your child! File for full custody?
I have been writing down a long list of things W has done around the kids for potential custody battles. I even put down things like "missed/skipped daughters ballet practice", " served unvaried food" (example only pasta and butter).
I have a list (and some videos). My state demands a lot to move much off of 50/50.