A similar effect happens to vampires in Vampire: The Requiem. As they get older and lose touch with their Humanity, it gets harder for them to interact nicely with mortals. They forget to do things like blink, breathe, vary their vocal inflections or send off the other signals that humans unconsciously do without thinking. Even if they do make an effort to do all these things, vampires that have lost enough Humanity will appear like walking, talking corpses:
A Kindred with low Humanity can put great effort into acting like a living person. He can force himself to breathe and remind himself to blink now and then... but he can't fake that subtle, unconscious dance of nonverbal interaction. Mortals soon pick up on this. They cannot consciously spot the problem, but their instincts tell them that something is very wrong and they should get away.
The Nosferatu clan gets this all the time, no matter how high their Humanity. Some of them look just plain ugly, but others might look perfectly normal... but when they interact with other people, they may carry about them the sterile scent of a hospital ward, or a gaze like they want to see what the other person's guts look like. They always carry the idea that there's something wrong directly centered on them. There's even an entire bloodline devoted to inverting the curse by bathing in blood to improve their appearance... and even then, it doesn't work, because they become too beautiful to be anything human.
This was also the case in the Old World of Darkness. A vampire had to spend blood in order to mimic a human, and the cost rose quickly with a falling humanity. Additionally, skin tones and monstrous features became more and more pronounced as a vampire forsook humanity, and alternate moral paths which abandoned humanity made vampires unable to spend blood to mimic human functions.
Elan in Dungeons & Dragons have this. Their skin is too perfect, their hair too red. This is because they are aberrations that merely look human. This also justifies their -2 Charisma.
Likewise, changelings in Pathfinder (the Half-Human Hybrid offspring of hags). They're Always Female and conventionally attractive, but their otherworldly demeanor and minor deformities (like heterochromia) make them disconcerting to be around, imposing a penalty to Charisma.
"Uncanny Valley" is the name of a trait in Eclipse Phase that can be taken in exchange for Customization Points. It means that your character's body (or "morph," in the game's lingo) is designed to look human (and not like a spider robot or an evolved monkey or something) and doesn't quite make the grade. It has the effect of giving a -10 penalty on all social interactions with humans.
According to the fluff, the MachineOrthodoxy invokes this. When a captured prisoner is compleated their face is often left just intact enough that their former comrades can recognize them.
The card Fleshmad Steed's flavor text sums up this trope pretty well.
More disturbing than the unknown is a distortion of the familiar.
Mortasheen has the Lester, a monster designed to look like humans and infiltrate their societies. But, because of "some imperceptibly trivial flaw in the monsters camouflage," humans immediately recognize it as a fake.
In-universe, Blanks, people who are born without a soul. While they look and act like completely regular people, other people are subconsciously aware of their lack of a soul, and the sheer unnaturalness of it causes them to instinctively hate and fear Blanks. Even if being a Blank makes you immune to the Warp, it won't do much good when you get lynched by an angry mob.
Cullexus Assassins, recruited exclusively from the aforementioned Blanks, literally weaponize this by amplifying their Uncanny-ness into a tangible and very harmful blast. They also literally suck out the warp energy of nearby psykers and a single punch from one of them can utterly destroy the victim's soul.
This is effectively what gives away upgraded Manei Domini agents in BattleTech. They are cyborgs who have been given a variety of implants, most of which are subdermal in nature. In theory any one high-quality implant is largely hidden, as is the case for standard characters who may need them. Too many, however, and they become subtly inhuman—agents with subdermal armor or strength enhancers have bodies that look more like a Rob Liefeld drawing than anything human, while agents with secret bio-weapon upgrades have odd patterns of head movement and speech because the location of the implants in their throats and mouths makes them sound 'off.' The tabletop roleplaying game Mechwarrior reflects this by forcing such affected characters to take unavoidable flaws that reduce their Charisma rating and make it harder to interact with others. This is helped by the fact that Manei Domini tend not to be the sanest of individuals to begin with, and all the cybernetic implants they receive frequently cause furtherSanity Slippage.