Put it this way, Ss06.. if you end up D, and I hope that doesn't happen for you, are you going to let him hang out on your weekends simply because he misses her so much? That's not typically the way D works.. maybe he needs a reality check.
That said, I'm not a parent, so that's just an unqualified opinion.