My STBX also hammers the point home that I am not "fair" becuase I don't ask the kids what they want, don't let them provide any input. For example, I signed up kids for scouting, (STBX: You didn't ask the kids and they don't wan't to do Scouts, Soccer, Summer Camp option A, B or C). "You call all the shots!" is what I hear over and over and over.
But I am the parent, and I will be confident about how long is appropriate at or near bedtime.
So you have boundaries with your kids - GOOD
I think insisting they do extra stuff like scouts is a GOOD thing. Having a good MALE role model is something all kids should have, so in your particular case I think it is even more important to pursue that with the kids. No matter what your relationship is with their other mother.