I know down deep in my heart that IF she was out of the picture we would be fine.
Doubt it. I doubt it because the OW is not the problem, right? The problem is the relationship you had with H, and so those are the issues that need to be resolved.
There are plenty of WASs who don't reconcile even after their A is over. In the meantime, the OW may be just the thing that hastens the end of their relationship, especially if you start to become the better option.