Listen to the master....Rob. Your h is trying to make you move out (as my w told me to last May...still here and getting better), with a D3!!!!
Stand you ground, detached but loving. If he wants out, tell him you will not stand in his way. Live yours and D3's life for the moment, leave him alone to stew in his own juices.
Do not walk...it's what he wants..he can then say 'Look, she left me...I did try.' If he had truly wanted to go, he would have gone.
Be polite, set the ground rules, but your home is your home; he is the visitor at the moment.