The thing is that YOU are not the one destroying the M by saying No. The S that runs around and has sex on the side is the one that is destroying it. Besides, what do you have that has taken all these years to build? Better take a good long look at it and make sure you've got what it takes for the long haul. If a man is "allowed" to have sex on the side and his W does nothing about it....then you can expect it to happen again and again in the future. And the very idea that man got mad at his W for telling his lover that they had sex! That takes the cake.
It is not about what you feel should work in your M. It is about doing the work that gets the right results. Do what works!