I'm as big of a "boundaries" guy as anyone on this forum, but I don't think you're in a position to set any (other than the usual ones like refusing to talk to her if she's shouting at you, or not tolerating her texting OM in front of you or something like that). Just the biggie of "I cannot live in an open marriage," which I assume she already knows.
You can't really lay out your dealbreakers until a wayward spouse ASKS you to take her back.
The mistake you made was a very, VERYcommon one around here, in that you LEAPT at the first opportunity she presented to you, and you scared her off with your neediness. A cautious "Hmmmm, I'm not sure how I feel about that. I'm afraid it's not that simple anymore," followed by continued distance (and maybe some dates, that SHE pursued), was called for.
Can you get back to that place again do you think?