Don't forget rejecting men for sex is about the worst thing you can do to a male. They are retreating to teach you a lesson I suspect.
Look after your own needs first.