I agree with the others. Perhaps letting it sit a bit before you answer will help take some of the emotional sting out of the initial reaction.
I f it were me, i would stay away from anything involving the word "fairness" - that's emotional bait. It plays into the ridiculousness of what they call "time of possession" in football. My F and M did this for 20 years after their D.
If I were you, I would stick to the facts.
I'm sorry you feel that way. I was unaware of an agreement with your M. As per our original agreement, the changeover was to happen at 5pm. If 5 pm does not work for you, 4pm will be fine.
Try not to think of this compromise as a metaphor for your whole situation. It is only one interaction. You're dealing with someone who isn't making rational sense and is preprogrammed to blame you for it.
