An affair has legal, financial, emotional -- even medical (just check out today's development from MarriedCrazy) -- implications on a family.
So given those implications, the OP *should* be confronted as long as OP is M.
But those same implications are insufficient grounds for confronting OP -- perhaps to undermine the excitement associated with the secrecy -- if OP is single?
SO the legal, financial, emotional, and medical implications only matter in the case of OP's spouse?
No, those implications only mean that the other person's significant other should be exposed to -- not the other person confronted themselves. It is up to THEIR partner (and I would include both spouses and long-term Rs here) to do with the information what they wish, but at least they'll be doing so in an informed manner.