Well, if I were in your shoes, I would remind my h that this particular weekend is your weekend to do things w/your children. His "activities" should and can be done on his weekend visitation. If you start switching up visitation days, it could get very confusing and then you won't be able to make plans on your respective weekends w/the children. I would be nice and also businesslike and tell him to reschedule his "activities" for his schedule time period and then let it go.
Sit quietly, the answers will reveal themselves when you least expect them to. The past is gone, the present is a gift and you need to focus on today, allow the future to reveal itself when it is ready.