I'm not surprised you're furious, Dilly - I would be too. And I don't think getting his key back from him is unreasonable. He doesn't live there. When he came for the weekend he treated you with disrespect (there's no need to set ground rules for this kind of thing in advance - you expect normal adult behaviour and those expectations of him cleaning up after himself were totally reasonable). It is your home. He has his own home that you aren't allowed into.