clearly this issue needs to be discussed and finalized. i'd suggest to write down an schedule agreement, put it on paper so there are no gray areas. pick up times, what to do if one or the other needs to swap, swap agreement - one day for one day, or entire weekend, etc.
all on paper, signed by both if you feel this is necessary. verbal agreements allow confusion and can be denied.
have this all pre-arranged, and completely understood by all parties. and if need be, explain to children and get their input, they may be old enough and would most likely appreciate having say in their schedule.
this would also be the opportunity to schedule holidays, birthdays, days schools are closed, sick days, etc.
my agreement is set up with everything laid out, and for holidays we swap each year. odd years i have certain holidays, even years she does. christmas eve one year, christmas day the next. i have my daughter on my birthday and fathers day, she has her birthday and mothers day. i can plan out my schedule for the entire year.
anything that is done like this helps alleviates the aggravation you are going through now.
"In a ham and eggs breakfast, the hen is involved, but the pig is committed".