By all means, if you don't want to go to dinner, then cancel dinner. Let H and D go.
You deserve a break.
I don't think it's controlling or manipulative. You can be honest and just say that you need some time to yourself (which is distancing and therapeutic at the same time). Don't say it in a mean way -just a calm way.
