There are several kinds of barriers to participation in RPGs. I won't be touching on the kind that the GM (you) can't do anything about. This post is mostly about troubleshooting problems with new players. I might want to do a post on social contract soon. Here's what someone else said: http://rpgathenaeum.wordpress.com/2009/04/07/is-a-social-contract-right-for-your-dd-group/
Stated Barrier: I don't know these people that well.
Translation: I don't feel comfortable acting or sharing the fruits of my imagination with people I don't know and trust.
Problem: The new player feels vulnerable. Even if he plays your game, he will hold back.
Solution: Establish a norm of honest openness among your players. Don't let them explain how they get into character or play the game. Encourage them to ham it up and acknowledge how silly they appear. Remember that even one criticism could shut down the new player. Model hammy behavior by doing voices, embarrassing yourself, and generally showing trust by going out on a limb for yourself, too.
Stated Barrier: I feel out of my depth with this setting.
Translation: The other players seem to know a lot about the world this game is set in, and I feel constrained in my creativity. While they can invent ideas and interpret the world, I feel like I can't.
Solution: Bend the world to fit this player. Make it clear that there is no inviolable canon. If she says she's a Tremere who defected to the Sabbat but came back, don't explain all the many reasons that's not possible. Instead bend the setting to make that possible. Use the setting to make it more interesting, but not punitive or un-fun. Maybe she was a sleeper agent and doesn't know it yet! Maybe she was an experiment in breaking Vaulderie! To hell with the canon.
Note: This is one of the reasons that organized play -- be it One World By Night or Pathfinder Society -- has a greater barrier to entry than troupe play.
Stated Barrier: I don't get these rules.
Translation: The rules, or just the presence of rules, probably seems too complex for the character.
Solution: One easy solution is to just take away the rule book and handle all the system for them. Some games make this easier than others (though even in something as detailed as Pathfinder or 4e D&D, there's a class for this sort of player). A better solution is to start slow. Starting slow is good for pacing in general, but it also works for people shy about rules. If your sheet is a tangle of numbers and everything you try fails, you start to feel like you can't crack the code. If, instead, you tend to succeed at almost everything, you'll start exploring and trying crazy new things. If those don't work out, you don't feel like it's your fault for not understanding the system. There is a problem with starting slow: Poor party balance can prevent this solution from working. If you start off with easy rolls, to give a newbie player a 90% success rate, but the rest of the table has optimized sheets, they'll be pushing for insane stunts (because they can) and the newbie player will go back to feeling like he can't crack the code. Games where character build is important (Vampire LARP, Pathfinder) tend to experience this problem more often.
Stated Barrier: ...
Translation: This player is a wallflower or is feeling excluded.
Solution: Wallflowers, you can draw out by giving them opportunities and challenges. Reward their contribution, and make them feel like how they responded to the challenge or how they took advantage of an opportunity was a clever choice. Empower their contributions and they will contribute to the point where they're comfortable -- but usually not more. Some people are just naturally wallflowers. Another solution for wallflowers is to make sure your eager players know to step back. Step forward/step back is a facilitation technique used by professional meeting facilitators to remind participants that some people just talk more than others, and it's OK as long as they know to consciously step back and wait after they've finished.
Stated Barrier: I feel like they don't like me.
Translation: You may actually have to draw this one out of the player. They may appear as a wallflower for a while before expressing this to you, if ever! They feel excluded.
Solution: There isn't always a solution. If a player is feeling excluded, there's one of two problems. One, it can be because the other players are excluding them. This is common and typical human behavior toward someone you don't like. You may wish to talk to the opinion leaders among your other players to see why. They should tell you if they don't like the new player (see below!). Groups of friends don't always accept a newcomer, though, so it may be irreparable. You may just have to remove the new player. There should be no hard feelings. I have friends I wouldn't play golf with or talk about politics with or game with. They're perfectly cool people, but sometimes personalities clash! Alternately, it could be because the player feeling excluded perceives that he is being excluded and is afraid to participate -- but the other players like them. Usually it's the other players' behavior making the new player feel excluded inadvertently. Geeks can often geek out about things. That's what makes them geeks. Anyone "not in the know" will naturally feel excluded. If all your old players are Trekkies and the new player isn't, he might feel like all the Trek-geeking is intentional snubbing -- especially if it's a Star Trek RPG! In this case, you can get the other players to help the new player feel more included with social interactions outside of game, talk about personal lives, other hobbies, and other OOC chit chat. They may still be at a disadvantage in Trek talk, but they'll know it's not intentional snubbing now.
Stated Barrier: None. You don't like this new player.
Translation: Someone invited a new player you don't like.
Solution: Tell the person who invited the new player immediately -- or at least as soon as you realize that the new person is more unpleasant than you can handle (this might take a several sessions, unfortunately). You don't have to be mean. Here's a script. Feel free to use it without attribution:
"I don't think the new person and I get along. I don't feel good about having them at game. I'm sure they're nice and cool and all -- it's just we don't mesh."
NOTE TO MY PLAYERS AND CO-PLAYERS: I like you all! This is not about you.