I wonder if it could be one of two things: 1) he is mad at you for the times you did not give him sex, has detached from you, and decided at some point that sex with you was a dead issue (in other words, he is angry and wants some revenge), or 2) he is seeing someone else. Could this be possible?
Your idea that he hasn’t had much sex for many years and has therefore grown to not want it can only be true if he has been castrated (or has some other physical malady). My guess is that years of no sex has only made him want it more. But maybe not with you.
