Bright,

If you don't want him there until a certain time, I think it is appropriate for you to ask him to come later in the day. You are already going to be dealing with family stress, why pile more on worrying about when he is going to pop by.

Simply tell H you have guests in the morning and then give him a time that works for you. He might also appreciate that in case there are business things he wants to discuss with you. Maybe he would rather not be there when your family is there, either.

I don't like waiting, so I would set a time that would stress me out the least.

Absolutely agree with you. I wish to be done with all of this, too.