Have you ever had a completely unsatisfying sexual encounter? Unsatisfying, btw, can be on any level... I'm not just saying you have to O to have a satisfying encounter. Uhm, I'll answer 'yes' for you.
When we have repeated 'unsatisfying' sexual encounters... or any encounter that isn't really 'satisfying' us... our desire for that thing or encounter dries up.
You, for example, have no 'desire' to have sex with a woman who does not 'desire' you in return. That doesn't mean you don't experience a physical urge for sex.
In my current state of singlehood... I certainly experience physical desire to have sex... horniness, if you will... and I could, I'm sure... have lots and lots of sex... with a variety of people, I'm sure, to scratch that itch.
But my 'desire' to have sex with 'just anybody,' is extremely low. It would make my physical encounters... with 'just anyone,' very unsatisfying.