The anger is partially caused by infidelity... Spring, in After the Affair makes this comment :
Certainly while you are having the affair, and after its revealed you may feel more anger twoard your partner than guilt. That's because the two emotions are inversely related. The greater your anger, the less your guilt; the more venom you direct at your partner, the less you direct at yourself. The anger you experience may be legitimate response to the way your partner has treated you, or a defense against the guilt you feel for what you've done.
It is human nature to hate those whom you have injured (wrote Tacitus,, almost 2,000 years ago). What makes it hard to know the difference is that anger always feels justified -- its a basic characteristic of the emotion. 1.
1. Spring, After the Affair, pp 42 - 43.
I really wish people having affairs would read more... they would learn so much about their feelings... :P