I think at the time, I answered something like, "you're not beautiful in the conventional way, and would never win a beauty contest, but I still find your body captivating, mainly because you're inside."
You're right, "yes" would have been a better answer ... and not a dishonest one.
