I suspect what is happening, is that each time he comes back to you and the family, because there is no "no-contact" or "transparency" plans in place with you, he is re-contacting the OW and then he pulls away from you and gets all nasty again.
It's physiological. It's highly addictive. It even SHOWS UP ON CAT SCANS, believe it or not.
You're going to have to do what you can to get him BACK to the "I want to come back, what will it take?" stage with you, and at THAT time you can insist up on some firm no-contact and transparency boundaries. Until then, I'd suggest pulling WAAAAYYYYY back and detach and become mysterious and dim-to-dark with him.
