it sounds like you really want to stop and heal your M. this is good. i know its hard to do but think back to why you married H. more often you M him because you loved him and found everything exciting. what you are doing with the OM's is what you are missing between you and your H. if it is excitment, love, sex, being on the same page, etc is missing then you have to realize that you both moved away from where you were at.
take it from a man who found out his W has been had an A. it really does eat a man up inside. it hurts there is alot of pain, regret, alot of tears. i am dealing with this pain, as well as my W. sje fells horrible about A. hates to see me in pain. sound like your feelings are the same as my W's. again takr it from me stay away from the OM's and work on your M if that is what you want. be patiant with your H. you might have to grow a thick skin and not argue this only adds to trouble.