No. Don't say anything. Just tell him, "have fun" and go out and make plans on your own.
What Bond said ^^^^
And also, understand that your H made it very clear that he wants space. Specifically telling you that he would not go if you go is him pointing that out. Regardless of whether there are no client appointments.
