His past lovers matter because when you're feeling rejected by him, you have nothing but time to wonder why and look for reasons. I know how this goes from personal experience.
If it weren't a comparison to his other lovers, it would be something else--your mind doesn't want to accept that there isn't some explanation, and when you seize upon something really negative, like "I'm not as sexy as his old girlfriends," you're afraid to dismiss it because dismissing something you're afraid of feels like you're fooling yourself.
But . . . the trap in that is that it's easy to start believing the worst every time simply because you think it's the most honest approach.
The biggest difference between you and those other women, if they did actually have a more sexual relationship with him than you do (and it doesn't sound like you know that for sure) is probably simply that he didn't try to sustain it for 23 years with them.
I met my wife in college, and I was a virgin. She wasn't. I made her wait a long time before we went further than petting over clothes. I was bursting as much as she was, maybe more, but I wanted to be sure. She had never been with anyone who wanted to wait for any reason. A couple of short years after we began to have sex, her drive for it tapered off. Honestly, it happened before we were married, and I asked her to promise me that if we got married as planned, she would work on it with me. She didn't, for years, but I took her promise and we got married. Looking back, I think I was stupid to try to extract such a promise and stupid to accept it, but now that we're finally moving in the right direction, I can't be sorry that I did. For years, though, I agonized over this, and I analyzed endlessly why she'd been so sexual with her other boyfriends and nothing for me. It seemed unfair (and it was, but maybe not the way I thought.) It seemed like proof positive that she found them attractive and me unattractive. I felt like I was just her paycheck/handyman/roommate, while they had been her lovers. Unable to find a man who could satisfy her in bed and still be a loving husband, she had simply chosen me over them because having a loving husband who would buy her things and do things for her was more important to her than having a satisfying lover. I knew she loved me, in a way, but I vacillated between thinking she loved me like a brother and thinking she loved me like a cherished family pet.
None of that was true, and my attitude only made things worse. I went through all that analysis you see above and more without ever asking her whether I was on the right track. By the time we really talked through our sex-starved marriage and why it was not OK, I had years of her reasons and thoughts in my head, and most of them weren't real. They were the reasoning I'd come up with to explain to myself why my wife was punishing me by freezing me out of our marriage bed--why did she hate me so much? What had I done to be rejected and despised? She had actually been wondering, when she bothered to think about our lack of a sex life at all, what the big deal was and why I seemed to take a perfectly normal end of a sex life between two married people so personally. It wasn't exactly a healthy attitude, sure, but it wasn't the same as "Oh, how I wish my old boyfriend Fernando were here so I could remember what it is to be ravished by a Real Man!"
(Wow . . . I hit Preview and read through this post, and it brought the tears right back to my eyes. I'm not far from those days.)
