Purple darling you have every right to be peeved. Whether he is part owner or not you deserve the respect of H asking if he can go in. Would you have that freedom to go into his place? You need IMO need to set some boundaries, by him being gone he kinda lost that right in a way.
In my sitch, W knows that it's still her house but knows that she drew a line in the sand by leaving, at this point it is not her house. I have never made the boundary she just knows it's a respect thing. Heck I don't even know where she lives, never asked and she has never given it to me.
Boundaries you need to make this known for your own salvation.