I think the 48hr rule would work well here - it gives then time to squirm.
I don't think that you will make things any worse by waiting to see what they do but it may well be better to wait. If you do speak to OW's H you need to have some sort of idea going into it what you want or might gain out of talking to him. Odds on it will probably blow everything sky high. That would mean that you will probably have no control over anything - are you in a position where you think that is acceptable. Think about Theo's Sitch.
What does your husband do? Is there any other reason he could be there? Could he have been looking for books on M's?
Saffie
Saffie me 46 H 46 M in 1986 D20,D18,S16,D13 H's A 01/05 to 07/06 H recommitted to M 07/06 renewed vows 09/06 Going from strength to strength