The wearing of my clothes is indeed a huge compliment, even a turn-on for me. I remember early in our R, X would wear one of my Tshirts and boxers the morning after ML. That was hot.
Or I'd come home and she'd be wearing a flannel shirt or sweatshirt of mine. She told me that she would wear it to smell me and feel close to me while I was gone.
I think you're right, tho. He'd rather have arrived with you just wearing the socks!