DISCLAIMER: Possibly because I'm currently raising awareness to my "lack of boundaries", this is where I see some potential future problems.
If there is an existing agreement of visitation, and one parent goes all "willy-nilly" at the last minute and changes things without a mutual agreement beforehand, and there is not a boundary/consequence in place, this will be exactly what your future will look like over, and over.
Do not let this go. Be firm, but calm.
Might be too late to do much this time, but going forward, set the expectation. One example it could look like is, "when W relinquishes her visitation time and D goes back to Matt, W does not automatically get Matt's time in place of that, unless Matt agrees to the switch. If Matt does not agree, but D is brought back, then W loses both times". For instance.