Sexual desire and affection are very different things. Affection is something we have direct control over. Lust on the other hand is controlled by numerous factors such as length of time in relationship, stress, kids, physical well being, psycological well being etc....
I used to be after my W approx 4 times per week. Now, after all this R crap It's waaay down. Not sure why and I'm not to worried about it anymore. I've read lotsa books. Some of the info I gleaned from them indicate that sexual desire is like the tide with ebb and flo.
H's focus on this is a sign of imaturity. His comment that it was not a problem with OW deserved a kick to the nuts. Bottom line is what has he done to try to turn it around? What effort has he made.
If you can't see the effort, it won't work. It will take an effort on his part just to relieve your stress and worry. If he can do that, you will start to lighten-up and respond and the circle of lust can begin.
Take heart that this is not a quick process. Focus on small improvements. Personally, as long as H is giving dumb responses, I would not waste my time pumping him for information.
Also remember that most of us men are pigs. We use sex as a tool to achieve intimacy. Cut off the sex and we respond negatively.