Of course it's not your fault that it happened. He is being childish. He needs to take responsibility for his actions. The book After the Affair is very clear about this. You both need to read the chapter on blaming others for your affair. My husband tried this on me and I rejected it completely. The book helped. He doesn'[t dare say that one any more.
