"How to even talk to XW about this without getting frustrated at her response?"
By making a point beforehand to not get frustrated at her.
"I know she is going to react badly."
Mindreading. Fact is that you've been avoiding her because you're scared of her. Since you're losing that connection, you're forgetting how to talk to her. You need to find a way to re-establish that.
"I take them out on weekends rather than spending it at what is used to be my home because OM is around. I don't feel comfortable."
Stop being so afraid of a scumbag who is taking your place in YOUR family. Learn to be in the same place as him and stop avoiding him. If you disagree with something he is doing, bring it up. It doesn't need to be in anger, but you need to assert yourself.
M-43 W-40 2D - 9 and 5
Emotion, yet peace. Ignorance, yet knowledge. Passion, yet serenity. Chaos, yet harmony. Death, yet a new life.