JMO Guys, but the kids not only take his attention away from THEM, the all important OW, but they are a direct link and reminder of YOU. Just looking at his children will remind him of you, and that is the last thing she wants. Plus, they are bound to talk about "mom did this" or "Mom took us there".
It's not the kids fault, of course, just the immaturity and self-centeredness of both the WH and the OW. The OW in my case is breaking her neck to suck up to his boys, but not my girls, whom he adopted and raised as his own.