I can certainly understand how painful it is. If your W brought this up, this is obviously an important issue for her too.
This truly may explain alot of things going on in your M. We often unintentionally push the things we love the most away out of self defense. This is something my H has done to me....he was pushing me away before I could hurt him. I'm wondering now if perhaps you may not have subconciously been doing the same thing.