Sounds like you've had 4 years of remorse for your affair. Not that it does you any good now, but I wouldn't have told him any more details than he already knew. I put myself in your husband's shoes and it would be hard for me too.
But is that all there is to this? I wonder. he came back several times knowing you had an affair. Why does knowing more about it drive him out again? Did you not tell him that you had sex? If he knew enough of the details I can't see why he'd leave again unless there were other things he either liked about being single and independent, or there were other issues in the marriage that he was running from. Is it possible that he's interested in someone else? Don't put it past a guy to use this as an excuse to get out when he wants to have his own affair in a tit-for-tat way. Just mainly thinking out loud, but something really seems amiss. I know getting over affairs is difficult, but this seems a little excessive to me.
You cannot be lonely if you like the person you're alone with. Dr. Wayne Dyer
