Have you ever thought about rotating YOU and H at the apartment. You sleep at the apartment on the nights he has them. Many folks are doing this now. This way the kids stay in their home but you both have ample time and space with the kids. You both would have to be on board with it. Your H has the best of both worlds right now. Just saying.
On his weekend you sleep at the apartment. On your weekend he sleeps at the apartment. Same for during the week
I see your points though. If you are happy with the arrangement then great!
