The judge probably will allow the kids around OW. Pick your battles - if the stipulation is lifted, it is. It just is.
Does primary placement mean full custody? Does that affect how much child support he has to pay? If so, expect a fight.
I know it's hard, but try to separate out the business part of the divorce from the emotional part. I know you don't like his parenting style and that it galls you to think of OW being around your kids. But those are things you'll have little or no control over after the divorce.
Focus on a solution that gets you out of the house ASAP and on some kind of reasonable financial footing.