Really, he's not even slightly interested in the truth, he's interested in dragging you into an argument and "proving" to himself that you are to blame and he is the poor little victim.
Rob is brilliant!
I used to 'state my case' when H would bring up all my wrongdoings, he was NOT interested in hearing any of it. Nothing. If you do respond, make it short and sweet, mentioning that you have tried in the past, that you still don't want this, but aren't willing to work on anything (marriage, sex, etc) with a third person involved.
And yes, he is pathetically using your love for D to threaten you. Don't fear that, your D, no matter what she is told, knows the truth.
I hope he calms down. When are you supposed to move?