I think an appropriate boundary would be - I will not be in a relationship with you while you are in one with another man.
This ^^^^^^^ is Cadet's example of how a boundary could be stated (that would fit your stitch).
Okay, so you state to her what your boundary is. Then you don't bring up again. You don't remind her as if she has a memory problem. You just watch to see if she is honoring your boundary. The first time you see that boundary line crossed.....then you act. How will you enforce your word?
In the above example, you've stated what YOU won't do. Right? You won't be in a relationship with her. You did not say "marriage", but relationship. (This may seem like splitting hair, but it's an option). What are actions that show you are no longer in a relationship with her? Cadet gave examples for actions one could use, that show consequences to the WW's choice to dishonor her H's boundary. Will she see it as the consequences of her choices, or will she pull the old guilt & control cards? Well, you hope she sees that she brought on these consequences.....but she may not. Your focus is to protect your feelings. Therefore, you withdraw from any type of acts that signify you are in a relationship with her. Basically, you stop acting as if you are her H. Many LBH's find it to be very difficult b/c they want to persuade and prove to the W how good the MR could be again. However, it just doesn't work that way with WW's.
If a man says, "I will not live in an open marriage", then he needs to plan his actions, based on what he said. As long as he is living under the same roof with his wayward wife......he's living in an open marriage. It forces his hand to take action and file for D if his boundary is crossed.
Let me also clarify that the W is free to do whatever she chooses. She has two options. She can choose to honor his boundary and cease to contact OM, or she can choose to continue having a third person in the marriage. If she chooses to violate the H's boundary, then the next move should come from the H, by him following through with action. If he leaves the M, it will be the consequences of her freely choosing to bring a third party into the M, and dishonoring her H's boundary. She may, or may not, suffer from her H divorcing her. But his focus was on what he could not & would not tolerate in his life/MR.
I want to also bring up the subject of boundaries that concern your children. These days, it seems the law protects the WW's freedom, while the H stands there helpless. Check out the laws for your state and see what you can and can't do. If you are S/D, you can't tell her not to bring the kids around the OM. You can't tell her what she can and can't do with the kids.....if the law supports it. See what I mean? It limits the H's boundaries about his kids. My suggestion would be to talk to a lawyer who advocates fighting for fathers rights, before you start barking about what you won't allow around the kids. Yes, of course you want to protect them, and they need protecting. Just know where your words have power....and where your words are powerless. If you can't back it up.....don't say it.
It is not about what you feel should work in your M. It is about doing the work that gets the right results. Do what works!