I do not know what to do in that situation. If your only time with your kids is then, I know I could not sacrifice that time. I think there are much wiser minds on this forum who have been down this road. They may have answers or suggestions on what to do with setting boundaries. I know of people who added caveats like this to separation agreements. Not sure if it has worked.
