The issue with you going early for the kids:

Could it be that he is jealous?

Jealous that you can't wait to see them so you go early?

Jealous that he doesn't get to spend that quality time with them/you?

I know you have offered all this to him but just a thought.

Me 43
XH 45
M 2.7.88
Divorce 7.10.09
Kids D20,S17 & D15