Hopeful, I think that's a tricky area. My W has never said that I'm controlling.
On the one hand, perception is reality, so if your H feels you're being controlling, you need to pay attention to that complaint.
On the other hand, if H is using that as an excuse to be able to do whatever he wants and not be held accountable, that's not acceptable.
I don't think expecting your spouse to be loyal is controlling. Telling your spouse they may not contact OW/OM could be construed as controlling. I think the better way to phrase it is in terms of a boundary -- if you continue to contact OM/OW I can't stay in this relationship. In that context, they can do what they want, and you'll take action as appropriate for yourself.
Accuray
Married 18, Together 20, Now Divorced M: 48, W: 50, D: 18, S: 16, D: 12 Bomb Dropped (EA, D): 7/13/11 Start Reconcile: 8/15/11 Bomb Dropped (EA, D): 5/1/2014 (Divorced) In a New Relationship: 3/2015