I am not one to frivolously advocate divorce. In fact, I really only think it is proper in cases of physical violence or serial unfaithfulness. But, your husband seems to be serially unfaithful... If my W had multiple PAs, I don't think I could move past that.
