Leaving aside your H's crappy comments, I, personally, find nerdy women very desireable.
Here's an anecdote. I just got a couple new pairs of glasses, one pair of which are horn-rimmed and rather nerdy looking. I asked my W to try them on, just to see how she looked (she's a wire frame wearer, but usually sticks with contact lenses). She put my nerdy glasses on and I got excited just looking at her. I told her, "you better not get this style of glasses." She said, "why, because they look bad on me?" I said, "no, because I'd probably 'bother' you twice as often if you wore those."