Affairs aren't real relationships. Neither one of them were representing a whole person to each other, complete with the responsibilities and restrictions that "real" relationships entail. The fantasy (or as my H called it, GriM fairy tale (apologies to the brothers Grimm-)) is what is so hard to give up because then that means they were duped. They hurt their family for NOTHING.
Our H's were not rejecting us. They were trying to find something... anything... to make them feel that "this" isn't all there is when real life overwhelmed them and they feel like they failed. It wasn't about how he felt about the skank, it was how she made him see himself. If he made her happy, he wasn't a failure, it was you bringing him down, in his mind.
He already had the script (married too young, wrong reasons, bitch wife, doesn't understand me, blah blah blah) prepared in order to allow himself to cheat in the first place. Facing the world isn't as hard as facing you, his family, and himself and admitting he screwed up royally and needs forgiveness. He is with you because that's exactly where he needs to be.
An affair doesn't have to end a marriage if we don't make it more important than it was. That only stops us from moving forward and hurts US. Don't give her any power. Your marriage is none of her business.