I realize that there is a lot of pain for everyone involved.
When my husband and I first started dating, he stayed loyal to an old girlfriend despite the things she did try to ruin our relationship.
Now, I know that you are in a completely different situation. So forgive me if I'm projecting...but I'm going to throw this out there.
My husband (then boyfriend) expressed the same reasoning you did. But I still felt like he was being loyal to her instead of me. And that hurt...a lot.
Could your wife be feeling the same, despite your noble intentions?