Fashion sense matters, even for you, perhaps especially for you. I don't think you understand how important good style is, how much it changes your image in the eyes of your small-minded feminine acquaintances. You could complain, but girls have stopped complaining about how unfair it is that guys are visual, so just shut up and listen.
Women notice what men wear because their dress reflects the amount of time and money put into their clothes; since ancient times, fashion has marked how much wealth a man has and thus whether he is able to fulfill the traditional role of provider. I know that men don't always take on this role in our modern world, but it's somewhat instinctual. Because of this, you don't need a nice face to be considered attractive. If you don't believe me, take it from the American Psychology Association [really really interesting citation].
Oh darn. I was about to go into the importance of good shoes and good jeans, but I have to run...... be warned-- more posts on the same subject coming up [Bridget, I hope you're rooting me on....].