My advice is not to let the sex go. Don't use it as a weapon, such as punishing the spouse for an argument by withholding sex.