You want to accommodate him against your desires because you're a Nice Girl who thinks that you owe it to the universe to meet the needs of other people so that yours are fulfilled. Your problem right now is this resentment at your STBX, which is eating at you, so I think you owe it to him to decline the swap. He'll kick and scream, be ready for it, but that's because you're changing the rules, setting your boundaries. He'll adapt and respect you for it.
M39 D6 D3 (at S) S 2014-09 D 2016-09
"You can't start a fire sitting around, crying over a broken heart" - Bruce Springsteen.