Actually, I don't think what you said was so bad. I see it as an attempt to set the boundary, that your house is not his house. I think you need to be more direct about it, though. He isn't going to get it until you spell it out in black and white, and he is just going to keep driving you crazy!
