I'd talk to your IC about your brother. I agree the choice should not be contingent on W's possible reactions.
But, you are at an incredibly difficult moment in your life. Sharing a house with someone other than my spouse and child is a huge stress for me. Other people might find a sibling comforting. I love my sibs, but living with them would be difficult. So, I'm just suggesting that you carefully consider whether having brother with you would be a comfort and support, or stressful, or a mixture.
It is important that you take very good care of yourself right now. It is OK to do what is best for you right now. You need it and it is best for the kids.
Be gentle and compassionate toward WHG, then decide.
