Hi Casey, I have the same question about setting boundaries about the house, although my h doesn't get on my nerves (or try to be physical with me) the way yours does. Can you set up a schedule and if he wants to be in the house, you could arrange to go out? If you have a set schedule, then at least you would know when he'd be there so you could make other plans, or at least brace yourself for the interactions.
I also agree with you about it not being fair that h has a place to go off to when he wants alone time, and I'm here all the time, with nowhere to escape to!