What bothers me, courts, is that you still get in a tizzy every time H says something about moving back in. Just because he mentions it doesn't mean you have to agree or even respond.
I thought you were going to be gone when H comes over to spend time with D3. You need to take a step back and put the focus on you, not on H. Stop letting him control this sitch. It's your life, you make the decisions about what you want.