I totally get that. But you have no say in where he lives.
He is entitled (legally) to a specific amount of time with them...
I was a single mom for a long long time, to two little guys too. It was tough and it was not how I planned it either but it is what it is. And, I am a damn good mom. I made a lot of mistakes but it was never for lack of love and my boys know that no matter what, they can count on me to always try my hardest. They are teenagers now and I am with a great guy. Their dad never paid child support or saw them at all...
it is what it is
I stopped expecting fair a long time ago.
Once you let go of the idea that things aren't fair, you can open yourself up to seeing a lot of other opportunities.