On the other hand - remember he emotional Aikido. Not giving your ex a handhold to grab, not resisting things that you're not likely to win on, and letting him then have to deal with it - is likely to end up with you getting S2 MORE, not LESS. If your ex sees it as getting a rise out of you, he'll continuing pressing it. If there's not drama and it just ends up being an inconvenience to him to have S2 that much, soon enough you'll see him not taking S2 for all the time he has (in fact that's started already, correct?). I imagine the amount that Ex takes him will fluctuate based on whether he has a girlfriend at the time or not. He'll want to show the girlfriend what a "great dad" he is. His interest in having S2 will fall off when no ones watching.
Also, remember, Xmas is an opportunity for S2 to develop his relationship with his grandparents on that side. Unless they themselves are dangerous, it's generally good for a kid to have all the grandparents he can.