I am LD, xw was HD, somewhere in our fourties, she went to HD from and I declined into LD compared to her. I read this is one of natures cruel jokes. Which I believe in.
You mention he fulfils your other needs, so I guess your love tank is normally full.
One thing that made it worse is. XW wanted to sex a lot longer than I could deliver, she mentioned her girlfreind and her H made love 2 1/2 hours. So in my case I avoided it even more, because I felt I was not delivering.
I don't know the solution, just something to think about.