I felt that societal disapproval too, May. I didn't confide in tons of people about my H'd behaviour, and many people didn't even know he was living elsewhere for all those months. There are very very very few people who know about his A, and the ones that do know very little of the detail - only that it happened and it wasn't physical when I caught him. And I did feel that shame or worry when he came back - I even hesitated to tell my IC for a few weeks, and I've been seeing her for nearly four years now!
It actually helped me to work with that feeling. I was so hampered in making boundaries, deciding for myself, taking healthy action and saying 'no' by a fear of what H would do, what he would think, how it would affect him, how he would feel about it. I struggled to be firm with the kids because of my worry about what they would think and feel about it when they looked back as adults. And now, now I couldn't even decide if I wanted my marriage or not out of fear of what other people would think. It wasn't a HUGE part of the mix, but it was a significant part of the mix that was holding me back.
I worked on that - first with my H - my need for his approval is nearly nil now. I like it when I have it, I prefer to have it, and I weigh it heavily if I am making a decision that is about the two of us. But I don't NEED it and if he doesn't give it an and has feelings about that, I decline the opportunity to deal with them for him. Similarly, I don't need to be my children's favourite person every single day either. My judgement is more important than their on several key matters: I am the parent. And I am the expert on my marriage over and above any judgements my friends might make - they are free to make them, and I'm free to decline to hear it or factor it into my decision making process.
I feel so much freer and lighter because of this. I actually let people closer to me than I used to, because I am not afraid of being so smothered by my fear of their opinions that I have to obey whatever I imagine they want me to do. And it has taken a lot of the 'work' of being married off my plate and put it back onto my H's plate, where it belongs.
I don't think you have as big a problem with his as I did, May. I am probably still behind you in this aspect of my development, and still have much room for improvement. But I think letting go of the burden of taking care of other people's opinions about me and my business has been such a lovely 180 to make.