I think that people tend to forget about most of the affairs that go bust. There's no real significance to the failure of an affair in people's lives and nothing to remind anyone about the affair. However, when an affair turns into a long term marriage, well then you have a "successful" affair front and center where everyone can see it. The unsuccessful affairs just fade away. Most of the time we probably don't even know an "unsuccessful" affair occurred in the first place.