I understand how strange it can feel to spend time with the in-laws without h, I spent a lot of time with mine (all differing sections too) over the summer and actually found it to be more comfortable that way...then I was not so focussed on what h was doing and could just relax and enjoy their company. in-fact I found that I was spending much more time than normal with them simply because the invites we used to always get I would have to check with h's availability but over the summer if there was an invite it was me and the kids so I could accept more. seems h nocticed it too but didn't realize the true reason for the "extra" time...any way I'm rambling now..
yes h needs his space, let him have it and with time he may realize that the "space" he is seeking does not have to involve so much space from you.