I hope you had a good Christmas. Gosh I have just caught up with your sitch and it's my situation to a T. I'm wondering if your H isn't my H's brother as they are so much alike in their behaviour ( minus affair for your H).
I perfectly understand your fears as I had the same but at one point your H needs to be accountable for being a father and he needs to learn how to be a dad. At the beginning I have let H have free reigns on when he could pick up/ dropped kids out of fear that I was a bad mother and I hav realised that it was only fear of what H and kids would think about me. I changed when one time I had made plans (and was really looking forward it) and H tried to change his time last minute. My heart raced when I said no and I thought I'd die but I didn't. I also noticed that H was trying it on like he did with my SD's mother as she would always dropped everything to accommodate H, but it was only working one way (his way!). So I knew that I couldn't be like her, H needed to take responsibility for his kids and deal with them like an adult when he has them. So far I can say he is doing well and that is good for my kids.
What I'm trying to say is that if you don't stand your ground, you are not respecting yourself and your right to do things you like and have fun. What could be the worst outcome? It's out of your control if H has your S that day. It's his problem.