What if I had sex with you and then 2 minutes later I wanted sex again and you said "Sorry, Mojo, not feeling the desire." Should I feel hurt? This is, of course, a ridiculous example but most sane people would say that I shouldn't feel hurt because there would be an obvious reason for your lack of desire that had nothing to do with my desirability. Why does it make a difference if the reason is not so obvious or even reasonable?
I don't know if you should feel hurt or not, and my reasoning might have to do with how I feel about your desirability. You might be very desirable to me when I am horny, but afterward, maybe not.
The question I think is not whether you should or should not be hurt, since that is only theoretical, but rather would you or would you not be hurt, because that is what is, and what you have to deal with.