I think you should follow what agreement you have with your wife regarding when you have the kids, trying to get more time with him because that is what he is asking might look bad on your part like you are trying to keep him away from his mother.
Her time with the kids is her time and your time is yours, how the kids are during that time or how they act isn't a factor, unless there is abuse.
Thats just my input
Me late 30's W mid 30's T 15, M 10 S4, S7 ILYBNILWY June 2015 In house S July 2015 W rings off Oct 2015 My ring off Feb 2015 Separate houses June 2016