There's nothing that says you have to go IN his apartment. You can do your drop-off's and pick-up's from the driveway or the front porch, if you want.
And I hope you know me well enough by now, that if your husband's work was near YOUR place, and the ONLY reason he was living in this town was because of OW, there is NO WAY IN HELL I'd advise you to go all the way there.