"His joke was insensitive for sure. He's threatening to leave you and your three kids."
I can't stress this enough. From a guy's POV, it was just a joke and you're reading too much into it. PLUS he did not threaten to leave you and the kids. He said (paraphrasing) that if he didn't want your M he would have been gone already.
You're going through "analysis paralysis" whereby you're breaking down every single thing that he's doing. If you keep doing this, THAT is what's unattractive and is going to drive him away. No one wants to live under another person's microscope.
You keep saying that he's "miserable". That's YOUR interpretation. He could have had a lousy day at work or something might have come up that he's trying to process. IT doesn't have to always be about the M. That's why I stressed that when you hug him you have to tell him that you can see something is bothering him and that while he may not want to talk to you now, you're there for him. Just the hug isn't enough.
M-43 W-40 2D - 9 and 5
Emotion, yet peace. Ignorance, yet knowledge. Passion, yet serenity. Chaos, yet harmony. Death, yet a new life.
