Don't choose to say no just to "prove" something to HER. That is the opposite of detachment...

Make your choice based on what's best for you and your son. If that includes shifting your schedule an hour on your days, then why NOT do it?