Good for you for holding your head high around her!! She is the one that has shame to feel.
LWB,
You do need to work on what you feel on the inside. Right now it hurts but your confidence will come in time, I promise. Everyone has to heal in their own time and in their own way. But from what I read on here you are a hard working woman, a fantastic mother, responsible, and I envy your strength. He will be the one losing IF he walks away. There isn't a lot of women who have the kind of morals that you have. She is only a fantasy and an escape from the crappy way he feels about himself. I think sometimes for men it is the hardest for them to come to the woman they love and respect more then anything in the world. Because they fear they will be seen as weak. Then they do the horrid things like try and push us away and put blame on us.