While I'm not into swinging, it makes sense to me that there should be rules involved in participating in that lifestyle - chief of them, respect for the willingness of each partner to take part - if he cannot respect your decision, the problem is not the swinging, it goes much deeper than that
For a non-swinger, you nailed it, I'm impressed. There are rules, and some couples have more than others. No. 1 rule for everybody is "No means No". H doesn't want to accept that from me as a permanent answer. But, he claims this is a minor issue... It's not to me.