Affiars ARE taboo. I think we live in such permissive times, and hold personal freedom so dear to our culture, that affairs slip through the moral net (also today we like to talk so much more about rights, than morals, as if morals were an outdated notion).
Affairs also freak people out because it causes them to take a stand - to be for or against, and this requires people to be sure of their moral grounding/position. Today though, mutlplicity of viewpoints is celebrated, not village mentality where right and wrong are distinct.
One friend of ours said, "I was reluctant to criticise WAH because I thought , omg what if I do the same thing one day and I get judged?". !!