LnL- you're missing the point here. H is there to spend the night - spending time with D - you tell him you have a "date" that evening and will be back later. He doesn't need a key - he's going to be watching D at your home while you're on your "date". He's going to (finally) get a chance to wonder what YOU are doing without him. He's going to see that you are taking him at his word and NOT reading anything into him being there. And if you don't give him much warning, I doubt very much that he would invite an OW over when you'll be back at some indeterminate time later in the evening.
More likely, he'll wonder about the aftershave on your clothes and the alcohol on your breath and the sexy dress you wore out the door!
