I know it is inappropriate due to the fact that he had fallen in love with his woman while having an affair with her & he is not setting up healthy boundaries to keep another affair from happening.... and I am totally planning to state all this to him. But am I willing to leave if it is just them chatting? Am I willing to stay in the relationship w/ the suspicion over my head that it could turn into an all out affair at any point (& I may or may not find out about it).
But if you clearly state that this particular relationship makes you very uncomfortable, and he chooses to maintain that relationship in spite of your discomfort, you set yourself up for a lot of misery, and he establishes that he's an accident waiting to happen...
It's also setting a marital precedent for "My spouse feels VERY strongly about this, and yet I don't care and I'm going to continue to do this regardless of her wishes." That's a huge problem, IMHO -- at least it would be with me.
I'll tell you a personal story, as a way of example. My wife and I are LONG since reconciled since she had her affair back in 2007, and we put everything back together in 2007 - 2009. It never even comes up anymore, I maintain no ongoing intel on her (and haven't for years), and things are very good. An old college FEMALE friend of mine messaged me on FB, and was trying to meet when she came to Florida. Might've been totally innocent, might not have been. I responded to her that "Sounds like fun -- my wife and I would love to meet you for lunch or something!" and she even seemed okay with that. So again, probably nothing.
But I told my wife about it, as I'm always sensitive to mutual transparency with each other, and considering what we went through I'm always cautious of playing with any kind of fire. The fetching Mrs. Starsky said she'd feel better if I unfriended the person on FB.
I took all of five seconds to decide to do that, and to let her know that it'd been done. She said "thank you, I appreciate it." The old college friend probably thought me rude, but my relationship with my WIFE is more important to me. If something really bothers her, I'm going to be sensitive to that, and I wouldn't want to remain in a marriage where that wasn't the mutual case.