Back on topic; I agree with most posters here.
We are a bit strange. A lot of people have hinted or said we are outcasts for the most part. Generally, gamers aren't the most popular people in school. Many are shy or have other problems (antisocial, anti-shower, anti something) and we'd prefer to discuss topics of our latest character/world/adventure/campaign/etc. Unless a person is into that, they might just not want to talk to you if that's all you talk about. That's not to say I think gamers have a one track mind on just gaming. But most people in the hobby are fanatics; at least the ones that the general public title: Gamer. And why wouldn't others think PnP games are odd?
1) It's been said already, but it is very true- RPG's are played in seclusion away from people. Ignorance and the unknown think the worst.
2) Ditto on the "there's a high percentage of stinky gamers."
3) A lot didn't have girlfriends. I say didn't because the group is changing. More women are involved in our games than a decade ago, and there was a fraction of women that played games before that. Honestly I think the internet helped gaming immensely.
4) Gamers spend their weekends indoors, not playing or watching sports, going to the mall, partying, or studying. Thus, most people are going to think, "what is wrong with those kids?" There must be something up with that RPG game. Heck, when I was a kid, I was outside playing or reading my religious text." (For the record, I am a Christian, but I have to roll my eyes at those types of remarks.)
5) Gamers don't generally garner the most healthy (or the most attractive) bunch of individuals. Many gamers don't exercise (much), and the big type of food is junk food (pizza, chips, soda, etc)
Now while that's the stigma, it may not be the case in all games, of course. However, in many of the games or conventions I have been in, I have seen this overall.