It is not a good idea for the reasons you mention above, but also because it needs to be your H decision.
He needs to choose to be with you (and just you) because that is what he wants. Not just because OW is not an option at the moment. The problem needs to be fixed with him, not with her. There will always be temptation out there and you can't fight with all of the tempataion - your H is the only one that can do that.
