I myself thought of another: When you say you hung at home and he says "Why?"....could you not answer "Because I felt like it"....if he persists with "Why?" You can say "Um, what part of "because I felt like it" (or whatever reason you've given) don't you understand?"
I also like the idea of turning it on him and asking why he wants to know....and if he hedges YOU can be the one with the "Whys?"