Put another way, I don't think wives "owe it" to their husbands to have sex with them, or vice-vesa.
I think wives do "owe it" to some degree. It's an obligation by default, because no wife has the "right" to expect their husband to remain faithful by being celibate. And so if extramarital affairs are against their morals, the wife is directly "forcing" the family to split up. You can only say the wife does not owe it if she does not owe her family any effort to preserve the family.
Of all the emotional, financial, etc., needs spouses fill for each other, sex is unique in that ONLY your spouse can fill the need for sex. Friendships, financial assistance, help around the house, can all be provided occasionally by friends or relatives. But not sex.
So I think that refusing sex with your spouse is the equivalent of cheating. In both cases, you have broken the implicit agreement to be exclusively sexual. You don't get to redefine "faithful" to include "no sex". Otherwise, I can logically say that because I'm not having sex with anybody, I'm faithful to every person in the country! Being "faithful" means YOU ARE ACTUALLY HAVING SEX. If you have no sex, you are in fact unfaithful, or at least NOT sexually faithful.
