If he wants to kids to be in this activity, then he will need to figure out how to take them on the days he has them. Let me ask this, what does he do w/the kids when they are w/him? Does he work late or work from home? I wouldn't set myself up to be the taxi for taking the kids all of the time.
If he can't work out the schedule w/you, then the kids don't go to that activity. Why tie up your free time when they are w/him?
Bottom line, he either works out the schedule to ensure that they get there and back on his days of visitation or they don't go.
Sit quietly, the answers will reveal themselves when you least expect them to. The past is gone, the present is a gift and you need to focus on today, allow the future to reveal itself when it is ready.