I can only speak for myself, but I was so frustrated from years of my H's physical distance before I had definite proof of his infidelity. His betrayal seemed that much more hurtful, to think that everything I had wanted so deeply he had been giving to someone else.
And then came the awful thoughts of how hot their sex must have (could have?) been. GAH.
It didn't take me long, however, to decide that I wanted to start a new experience with him. To get rid of the shadow of the OW, for me, meant rekindling that physical relationship with him. Doing new things, creating new memories...you get the idea. I want him to be attracted to me, to WANT me. I never let an opportunity pass without making sure he knows I am desirable and fun.
It took him awhile to be ready, but since then, things have been much better.
I know it's hard to justify that with the pain they've put us through. But we do have to put the pain aside and start back up the mountain at some point. Better when they are willing than trying to gain more ground when they have given up hope.