I know it is hard to but you've got to stop letting stuff like this get to you and letting your mind pan out a scenario which may, or may NOT be the truth. Me, and a lot of other here, no precisely how you feel and the torture that you put yourself through. It just isn't worth it. Whether you are going to work on you marriage or punt it, thought processes like that do nothing but bring pain to YOU, not ANYBODY else. I know it is difficult to realize and bring into perspective, but you need to discover ways that you can deal with thoughts like this and drive them from your mind.
For me it was many things. X had an affair with a teacher at my kids school, so everytime I was there I had a million thoughts going through my brain as to who knew, who didn't, etc, etc. After that was an affair where she got busted out publicly. I discovered where they had been meeting, pay phones they had been using, all sorts of clandestine crap. For the longest I let this stuff get the better of me and it was literally painful to drive down the road, go to soccer games, go to school, on and on. FINALLY, and I mean it took some work, I quit ALLOWING these thoughts to consume me. What he did has NOTHING to do with you and people are NOT looking at you with pity. I promise you that 99% of the things you think may have happened didn't, and even if they did, what does it matter. YOU didn't do it and it DOES NOT reflect upon you.
I coached the wife of my X's OM through everything and they are still together and trying to make it work out. She had hostility and anger out the whazoo and said many of the things that you do. Right now, they are well along in piecing and things are looking good.
Chin up girl... take a step back, rethink the whole DB philosophy and make sure that you are implementing. I'm not hearing that you're done, and any marriage w/ kids is worth saving. It can be done. Even if your choice is not to save it, self inflicted punishment like you described above is something you have to learn to deal with right now, for your own sanity. I feel ya so much on this one...