It is indeed super tricky with kid schedules. You need to stay connected somehow. M and I are on the same weekend schedule, which helps. I only have one night a week to myself and we make a night we can see each other. D11 can stay home alone for a period of time which she does after school, then she will stay home while I go to the gym for an hour less than a mile away from home. I have never left her alone to go see him, even in town. She is an only child and not a big fan of being alone unless her best friend is over.
It is a puzzle that needs to be put together by both parties, but you can cross that bridge when you come to it.