You may not be able to get a place for you and the kids, IDK. I don't see it as weakness, myself. I see it as a man who is fed up with her affairs and is taking his kids and getting out of that house, if she intends to stay there. However, everyone does not see it this way, and believes the man should stay, regardless.
Do you want to hang around there while she openly conducts her affair......? Isn't that like rubbing it in your face? This is what I meant about talking to the lawyer about your options. I didn't mean filing for the divorce, but about you leaving the house, protecting your finances, etc. I don't know the legal ramifications, so check with him first.
It is not about what you feel should work in your M. It is about doing the work that gets the right results. Do what works!