If affording the place is a problem, then you are either unemployed or underemployed.
To get better job/pay, you will need your self-respect back.
You won't get your self-respect back cowering in fear outside of your own home.
These women (including your own mother) are advising you as they are because they don't think you can pull this thing off. And don't get me wrong -- do not move back in if it's not part of an overall plan.
But make no mistake -- it needs to be Step #1 of any such plan.