It is emphatically NOT YOUR FAULT that he had an affair! That is HIS choice, and it was WRONG. Hell, if it was some one-night-stand, it'd STILL be wrong, but he's knowingly and selfishly made this decision FOUR SEPARATE TIMES.
Yes, you are BOTH responsible for the state of your marriage. But that gave him NO right to commit adultery, and you need to tell yourself that.