Cathy - I hate to disillusion you, but - - - RUN!!! RUN as fast as you CAN!!!!
I'm sorry that you spent the last five years on this guy, who clearly is pathological and should be attending some kind of 12 step program (SAA or SLAA).
This isn't a case of a long marriage with the late onset of MLC and affairs - in which case one might be justified in trying to salvage the marriage. This is a pathologically disturbed person whose behavior began very early in the relationship. Either he's a sex addict, or a narcissist, or has some other kind of serious pathology.
Do you know anything firsthand from his ex about why they split up (not what HE tells you, but what SHE says happened?). I'd be willing to bet he's done this before. Maybe he wasn't even really separated yet when he first contacted you.
I know it's upsetting to think about giving up a relationship that seemed to be good and that you have invested so much time in. But this is not a guy who is showing any evidence of doing what he SHOULD be doing to fix this (getting counseling to figure out why he does this, etc.)
Now - you said your divorce was 4 years ago and his is still ongoing - so you're not legally married, right? Is your home title in your name only? If so, consider the work and money he put into it as partial payment for the grief he's putting you through now, separate any joint accounts ASAP, check your credit report (to make sure he hasn't taken out cards in your name) and read the book The Sociopath Next Door.
It's not too late for you to find a real, honest relationship with a good man - but this guy is NOT the one.