I am not a therapist, nor do I play on one TV but as I read your post, the thing I take away from it is that BOTH of you have looked outside the relationship for something that was missing within it. Even the swinging thing, if it were just sex, that would be one thing (if it worked for both of you). But you write that he went to dinner and walked around with her etc...It seems he found someone to connect with, without the baggage, in ways that you two were no longer doing. Now it seems he runs to her whenever things get tough.
It seems you have the PMA/GAL down fairly well but what about your 180s. What do YOU think YOU could do differently to break the cycle and help him to feel loved?
Also, do you think it is possible that he has not, and is not receiving the love you were/are sending? You wrote you didn't know how to reassure him...I would suggest reading either His Needs, Her Needs or The 5 Love Languages. These books talk about loving in a way your spouse understands.
I am gonna go out on a limb here but I think your situation could work out. You might not have a husband who wants out so much as a husband who is really hurting and who "lost" his mate somewhere along the way and wants to stop hurting.
Patience, love, and detachment...yeah, it's way easier to type than to actually do.
Me 44 She 46 S13 D9 M18 T23 3 years DB'ing Successfully busted