A similar effect happens to vampires in Vampire: The Requiem. As they get older and lose touch with their Humanity, it gets harder for them to interact nicely with mortals. They forget to do things like blink, breathe, vary their vocal inflections or send off the other signals that humans unconsciously do without thinking. Even if they do make an effort to do all these things, vampires that have lost enough Humanity will appear like walking, talking corpses:
A Kindred with low Humanity can put great effort into acting like a living person. He can force himself to breathe and remind himself to blink now and then... but he can't fake that subtle, unconscious dance of nonverbal interaction. Mortals soon pick up on this. They cannot consciously spot the problem, but their instincts tell them that something is very wrong and they should get away.
The Nosferatu clan gets this all the time, no matter how high their Humanity. Some of them look just plain ugly, but others might look perfectly normal... but when they interact with other people, they may carry about them the sterile scent of a hospital ward, or a gaze like they want to see what the other person's guts look like. They always carry the idea that there's something wrong directly centered on them. There's even an entire bloodline devoted to inverting the curse by bathing in blood to improve their appearance... and even then, it doesn't work, because they become too beautiful to be anything human.
The Hot Chicks RPG doesn't address this so much in the game as it does with the artwork. The CG art instils the Uncanny Valley much of the time, that is when it doesn't fail to get that far with the fact that 2/3rds of the women's facial expressions look like they were modeled off blow-up dolls.
Elan in Dungeons & Dragons have this. Their skin is too perfect, their hair too red. This is because they are aberrations that merely look human. This also justifies their -2 Charisma.
Likewise, changelings in Pathfinder (the Half-Human Hybrid offspring of hags). They're Always Female and conventionally attractive, but their otherworldly demeanor and minor deformities (like heterochromia) make them disconcerting to be around, imposing a penalty to Charisma.
"Uncanny Valley" is the name of a trait in Eclipse Phase that can be taken in exchange for Customization Points. It means that your character's body (or "morph," in the game's lingo) is designed to look human (and not like a spider robot or an evolved monkey or something) and doesn't quite make the grade. It has the effect of giving a -10 penalty on all social interactions with humans.
According to the fluff, the MachineOrthodoxy invokes this. When a captured prisoner is compleated their face is often left just intact enough that their former comrades can recognize them.
The card Fleshmad Steed's flavor text sums up this trope pretty well.
More disturbing than the unknown is a distortion of the familiar.
Mortasheen has the Lester, a monster designed to look like humans and infiltrate their societies. But, because of "some imperceptibly trivial flaw in the monsters camouflage," humans immediately recognize it as a fake.
Warhammer 40,000 has Blanks, people who are born without a soul. While they look and act like completely regular people, other people are subconsciously aware of their lack of a soul, and the sheer unnaturalness of it causes them to instinctively hate and fear Blanks. Even if being a Blank makes you immune to the Warp, it won't do much good when you get lynched by an angry mob.