I think divorce is hard on kids. Your X isn't helping matters. You need to let go of your anger, too. I know how tough it is to do that with a toxic ex, but you can pray and be strong and do that I have faith in you.
I think if that happens again; you should remind your son that it's your time to be with him. My kids are very concerned (most kids I think) with being fair and stress that it would be unfair if one parent monopolized the time of the other parent. I would grab him if I needed to and carry him out of there (he's 5 not 15), and not let your X dawdle around hoping you'll change your mind. I'm sure it's tough for your S5 to be in that position; maybe like feeling like tug of war or something and he's the rope.
I would try to talk to him about it also. Like today or tomorrow, not the actual day. Your X may be saying something or he might have just had a bad day. It would be good to find out what's going on and talk about it. I do think, even if your X did something horrible, don't discuss it with your S, but instead maybe start documenting for future discussion with an L.
