An affair has legal, financial, emotional -- even medical (just check out today's development from MarriedCrazy) -- implications on a family.
So given those implications, the OP *should* be confronted as long as OP is M.
But those same implications are insufficient grounds for confronting OP -- perhaps to undermine the excitement associated with the secrecy -- if OP is single?
SO the legal, financial, emotional, and medical implications only matter in the case of OP's spouse?