Do you have any court issued orders yet? You may want to talk to a lawyer asap.
You have been going along with this verbal agreement for a while now. The fact that this is the way things have been and the kids are used to it, could work against you.
See a lawyer and at least ask some questions pertaining to your rights to your kids.
