Sue, if you feel you need to say a few more things, I don't see anything wrong with that, obviously do it calmly. H might not hear what you have to say or deny that its 'taking away' from your marriage. I see OW as filling a hole my H had, that he felt and partially that he created to justify it.
In my case, this is the 2nd A that OW has had. OW's H said he is done if this doesn't work, but we can all say that. I used to say that I would never put up with this, and look at me.
Don't be afraid of it sinking it. This will happen from time to time. Its ok.