Well, if you don't have concrete evidence, then you don't have evidence. Are you going to stop him from hanging out with the family because of your assumptions?
My point is: figure out your boundaries and, yes, "do-able" consequences if he crosses them. But don't come up with an actual boundary like that based only on your assumptions and insecurities. That would be weak. Does that make sense?
And I'll be straight with you: reading your comment about you not wanting him hanging around y'all if there's an OW came across a bit high-brow to me. You're not his mom or his moral authority. You're his W and the mother of his children. Even if you guys separate or end up D, he is still the father of your children, and he will be places with you guys all the time ... even if there's an OW in the picture. And I would caution you against ever using time with the kids as a consequence for his behavior. That smacks of you trying to punish him. That's not your job.
M: 40 H: 44 Married 14 years S11 & D6; D20 & D19 from previous M 2BDs/PAs, 8 years apart Piecing: April 2014