I don't buy into the idea that it takes two to make a marriage fail. I believe that it can be one person's fault.
Bunny, did you go into the marriage knowing your h was into swinging? Are you the one who changed the rules? I think not. I am convinced that you have done nothing to destroy your M.
Your h threw it onto the rocks when he coerced you into a risky, unhealthy lifestyle. You are simply trying to get yourself back into a healthy state of mind. You have been a loyal wife and a good mother.