Yes, the problem is that the one party (the husband or the wife) consents to the violation of the marriage.
OK, so maybe the wayward SPOUSE should be civilly or criminally liable? I mean, they have violated a contract, no? Which has led to financial hardship and emotional suffering of the betrayed spouse?
Hell, in most states it's not even grounds for divorce anymore. Unbelievable.