I think what has been holding me back is being afraid that her H will throw her out and she will run to my H.
This was one of my fears. Also, I was worried they (OW and my H) would move out and continue their A. Neither of these happened. I see now, that if it happened, I couldn't have controlled it. But back then I was paralyzed with fear, so I understand.
Your H is in a good place, wanting it to be over. My H was not in that shape.