As you know, I originally thought your wife was having an affair with your neighbor. I've changed my mind on that, but my opinion of their influence on her, and your family, is no less valid:
It's almost like she's addicted to their lifestyle, and she is rebelling against anything that stands in its way. They represent carefree, mindless escapism to her, and you and your children are viewed as anchors, holding her back and ruining all her fun.
Don't get me wrong, I still think she has had inappropriate feelings for HIM, and I think they've probably mutually even done some light, inappropriate THINGS (and I could even still be all wrong, and who knows, maybe it was a full-blown affair). But the important thing is that I am MORE CONVINCED THAN EVER (as it sounds like you are now, too) that these people are not the CAUSE of your problems, but they are certainly the immediate OBSTACLE TO SOLVING THEM.