Well, lots of us thought cheating was a deal-breaker - BEFORE we had to deal with it!

But in your case, Gwyn, there seems to be so much more here. You said yourself that you were ambivalent about marrying him because it didn't "feel right". Your D feels so strongly that he is a "pervert" that she states she would never leave a (future) child of hers alone with him (and she apparently didn't like him even before the A, right/). He hasn't been upfront with you about finances etc. - this is often a warning sign that they are hiding something (meals with oW, massage parlor bills, secret cell phones, gambling addiction, whatever).
This goes beyond a simple single "once-in-a-lifetime got-depressed and fell into an affair" thing, I think - and I think the reason you are having so much trouble moving forward with him is because, somewhere deep in your brain, you know this.

Ellie