Oh Coly. I just want to give you a big hug. I don't know if you have noticed that if there is a chance of reconciliation it usually only happens when the LBS lets go. You can love and let go. It's not easy, but you can.
Your exH is on guard not because you aren't polite and lovely when you spend time with him. Because he KNOWS you have expectations of you wanting something he can't give. Because he recognizes the cycle and knows the texts are going to come in. He DOES realize that you and D put your hearts on the line when he comes over. He knows it comes with expectation. And shame on him for not declining the invite knowing this and that he can't give you what you want.
I hate to see you go through this cycle over and over. I know how much it hurts you when he doesn't respond in the way you hope.
You have to let go for yourself, and D. Then who knows? But you are inflicting pain on yourself and you don't deserve to hurt like that.