You have no idea how many women who see guys in VS think it's sooo sweet that you are there. Women don't assume you are ogling the lingerie.
Heck, I've been known to walk up to a confused guy and ask if he needs some assistance picking something out for his wife/girlfriend (not flirting, just trying to help them out). Sometimes you guys can look so overwhelemed and uncomfortable in there LOL.