Not sure how they could? What do you worry might happen? Particularly if it's only shared as suspicion.... I think that seems safer. Since proof can usually be argued away (unless it's photos from a hotel room, kissing, etc...).
Although, I guess I can understand your hesitiation. I remember worrying about this too. I had no worries about contacting OW (and I was just seeking information and answers, no blame or anger initially), but OW's husband did worry me a bit.
There is no arriving, ever. It is all a continual becoming.