I can't believe I've missed your thread. I'm sorry you find yourself although you will find wonderful and supportive peeps here. These folks get it where *real life* peeps don't always understand.
In regards to your h's childhood, my h had a very loving and priviledged childhood. However, he struggled with very low self esteem and was held back twice in school. He would break down and cry as an adult at the ridicule and shame he felt. He always felt his older brother was better. So, while I'm not expert in MLC, difficult issues in childhood come in a variety of ways.
I can relate to much of what you say. My m was probably 3rd on my priority list and I too thought we were in this together. I thought we we would be together forever. I feel sort of silly thinking that now.)Work on things for you-to make you a better person and mom. We, too, struggled with lack of intimacy and that is something that you really can't appropriately show you h at this time IMHO.
I think you are doing a good job by staying out of his way. Respond and focus on not reacting. Just leave him alone. Be cordial and leave him be. Don't talk to his friends. Not your sandbox. Just focus on you and your little ones. Your kids are fun ages:-)
Hang in there! It gets much, much better.
3 kids BD 12/15/13 (IDKIILY. ) Rope dropped Cirque du Soleil style D final 9-9-14 "Some people are born on 3rd base and go through life thinking they hit a triple." Barry Switzer