You could. It depends: do you love him unconditionally? Does this change everything you knew about your marriage or about what kind of human being he is? What exactly makes it the deal-breaker? (I'm not saying it shouldn't be--but decisions made in haste can end in tears, so take your time and really think through what a divorce would mean for you, and what the thing is that you can't get past.)