This page shows that even things like the oft-ridiculed original Thought Eater can be this trope.
From D&D comes the vastly disturbing Atropal◊, an epic-level creature who is quite literally an undead god-fetus. As if that wasn't terrifying enough, even if you manage to kill the Atropal, bits of its sloughed off flesh can reanimate as less powerful but perhaps even more disgusting Atropal Scions◊.
Worst part - you don't need to kill it to make those. It can re-animate its own chunks.
A sourcebook called the "Book of Vile Darkness" is a whole vat of Nightmare Fuel, supplying the GM with ideas for various heinous acts, spells such as 'Mind Rape', 'Wall of Eyeballs', or 'Rapture of Rupture', as well as several monsters. One particular horror is an abyssal being who feeds by paralysing its victims and then digging through their bodies with a toothed feeder tendril. Bonus? The creature is naturally telepathic, so while it is munching on the victim's guts, everyone nearby (including the victim) will feel what they taste like.
The Book of Vile Darkness includes several "races of pure evil", who can also be this. Most prominently are the Vasharans and the Jerrans, evil versions of humans and halflings, respectively. Vasharans were a flawed first attempt at humanity that archdemons rescued from oblivion and further perfected; they are totally incapable of anything like pity, remorse, love or kindness. Jerrens are technically even more frightening: originally, they were just ordinary halflings. And then, to fight off the goblins they were at war with, they gleefully sank to levels that disgusted said goblins.
Let's not forget the sample character The Dread Emperor, who keeps children chained to him so that he can gain a minor benefit. Actually, most of the sample characters presented in the first chapter are pretty twisted, especially the elf child who happens to be a HALF-FIEND.
What about the Vargouilles, monsters that are essentially flocks of flying shrunken heads that kiss you and, if you aren't quickly healed, turn you into a flying shrunken head? Not even you as in all of you. Just your head sprouts wings and breaks off.
Planescape has the Vaath, which uses a tentacle with a sphincter-mouth on the end to burrow through your flesh and organs until it reaches your spine and severs it, paralysing you. Then it tortures you to death.
Lords of Madness: The Book of Aberrations is equally unpleasant. Let's consider: we have the aboleths (horrors from before the beginning of the universe, whose knowledge of the magical and psionic arts will actually drive you insane if you try and learn it), the mind flayers (pretty much the same thing, only from the future and who reproduce by sticking a tadpole in your brain, whereupon it eats your entire nervous system), the beholders (xenophobic and paranoid beings who are not only virtually immune to magic, but whose reproduction is basically vomiting up their children), the neogi (giant wolf-spider people who basically see the entire universe in terms of money, and will sell anything or anyone if they think it'll net them a profit — oh, and they can control minds), the tsochari (who are basically Always Chaotic EvilPod People, only they can inhabit people and slowly eat their insides), and the grell (floating tentacled brains who think the entire universe is just a smorgasboard). Oh, and there are about a dozen Eldritch Abomination gods like Mak Thuum Ngatha, Tharizdun, and Y'chak, many of which are utterly insane Expys of creatures from HP Lovecraft and would gladly crush the universe. These are all supposed to be happening on the same universe.
First off, a proper description: aboleths are giant tentacled leech-fish with varying amounts of eyes, usually in sets of three. Kinda like this◊. JUVENILES tend to be on the order of 25 feet tip-to-tail. They're also basically pure psionic energy made flesh, sometimes are able to fly, and each one has the combined genetic memory of all its predecessors. Depending on which edition they range from, the either hail from one of the deeper levels of the Abyss (the Chaotic Evil version of the afterlife; Hell in D&D is Lawful Evil) or the Far Realm, a.k.a. the lovely little hole in the order of the universe that spawns everything Lovecraftian. Their attacks tend to eat your memories, if not just completely Mind Rape you into being a thrall. They're also more-or-less immortal (as immortal as anything gets in the setting, anyway) and the earliest ones predate all other known life. Yes, a baby slug-looking Aboleth spawn wriggling in the mud (that's still twice as large as your biggest adventurer) has memories of the entire divine pantheon coming into being. The only, only up-side to them is that they're not really openly hostile unless attacked; it's just that their very PRESENCE tends to lead to horrible, horrible consequences to anything that isn't them, since their psyches and priorities don't exactly work on the same level as ours.
Mind flayers, who refer to themselves as illithids, are lavender-skinned, slimy Cthulhumanoids that eat brains. They are refugees from the end of time. They are ruled over by giant psychic brains with tentacles, formed out of their own corpses. Aboleths consider them terrifying because they don't have memories of Mind Flayers' origin.
With the possible exception of Atropus (possible, as even with Narm, he's pretty scary), most of Elder Evils qualify. Pandorym? It's an Eldritch Abomination from a reality perpendicular to the game world, and it's going to kill the gods and a fair number of planets too — and Obligatum VII, one of the representatives of universal law, wants to release it to fulfilla freaking contract. Father Llymic? He's a monster from the Far Realm who will transform the entire world into ice, transforming most of its inhabitants into hybrids of themselves and insects. Ragnorra? She's a giant wormlike sack of flesh that transforms into an (arguably even scarier) True Mother form, and she not only spawns an infinite stream of crimes against nature that make beholders and mind flayers look pleasant, but The Dragon is a maddened zenythri (for the uninitiated, these are the descendants of humans and beings of pure law) who has cut his lips off and replaced them with wriggling flukelike critters. Serthos? He's not so bad by himself, but his manifestation flooding the world with serpents, and he can can never really be defeated, as he's already dead. Kyuss, The Worm That Walks? Let's just say you may never look at a worm in the same way again. It's strongly advised you only read this book in dark rooms, so that you don't have to look at some of the pictures. And of course let's not forget Zargon, the last member of the devil race before the baatezu. Not only is he a vile-looking creature, but he creates a pool of poisonous slime wherever he goes. Plus, he is pretty much unkillable, as in the gods asked Asmodeus to take him down. Which even he couldn't do in the end, as he couldn't find the pit that you have to pitch Zargon's horn into to actually destroy him for good; I suppose it wouldn't hurt to mention that you have to do this within a DAY of taking him down, or that his horn makes him practically indestructible in its own right. Especially the golothoma◊, an eyeball-spider-serpent-thing that eats you with its shadow.
Dragon Magazine occasionally got into this too. One issue featured several spells designed to evoke that "insane asylum" feel, up to and including a lobotomy. About the only reason it's possible to sleep after seeing that is to consider the Shout Out in the opening flavour text features a "Dr Gregorian Ilhousen", a Hilarious in Hindsight moment. note Ilhousen's an NPC from the Ravenloft campaign setting and was most prominently featured in "The Nightmare Lands," a boxed set published in the 1990s, well before "House" ever existed.
The book in which Dr. Illhousen figures, The Nightmare Lands, is quite a nightmare fuel filled read in itself. Imagine you are a doctor who has made breathtaking advances in medical science related to insanity. Imagine you come across a pattern of similar nightmares in a string of patients. Imagine you realize the patients aren't psychotic, they really are being tormented by nightmarish incomprehensible beings that worm their way into your nightmares. Imagine when those beings notice you noticing them. Imagine theres no escape from them because the Nightmare Lands are a Domain controlled by these beings, who are collectively its Dark Lord, and that the Lands conjoin the Deep Ethereal, where all humans go when they sleep. Welcome to Dr. Illhousen's very sleepless and fun life.
The Wishing Imp is a magical statue-slash-Jackass Genie that follows you around and twists your wishes. And eats your soul if you don't make any— or if you do.
Illithids (see above) are scary enough. One of the Darklords of the place is an Illithid elder-brain. The stuff that torments it? Illithid Vampires.
Darkon is a very subtle kind of Nightmare Fuel. The place eats your memories and replaces them with memories of you having lived your life there your entire life. Which means that anyone there cannot, almost by definition, trust their own memory...
The Bleak House. Every room in the place is designed to unsettle the characters (and players) from a room made in shadowless monochrome with a permanent echoing spell to torches that give off monstrous shadowy. Every patient, orderly (and the characters) are made to dress in Grey robes, with ugly masks they can't take off, so you can't tell friend from foe, can't even recognize yourself. The kicker? Every night, when the characters are asleep, they're subjected to several different types of "Psychological Experiments" ranging from inducing new Phobias to purposely inducing Split Personalities and two words: Psychic Amputation. Not to mention the Warden has a nasty habit of letting you think you're getting away with a clever escape plan, before he swoops in at the very last second, and did I mention the Orderlies are dressed exactly like the patients and they're a type of vampire that drinks cerebral fluid.
One adventure features a trapped coffin that if some unfortunate victim gets stuck in it, but manages to escape before ten rounds, that they start acting and suffering the effects of vampirism for the next several days (but not actually turning into one). Which leads to the idea that the poor victim might end up 'being staked for the better of sapient-being-kind' before his friends realize the ruse. Or worse, imagine being the Paladin or Cleric (or a Ranger with special enemy Undead) - imagine their horror.
In the greater D&D cosmology, gods restrict themselves to their own domains, e.g. the gods of Toril have no influence on Krynn. Ergo, clerics from the Forgotten Realms have to find a new god if they end up in Dragonlance (which, philosophically, is a lot harder on a person than erasing "Mystara" and writing "Majere" in its place on your character sheet). That is not true for Ravenloft; while no true gods have power in Ravenloft (late 2E Vecna notwithstanding), which has its own pantheon of ill-defined deities, outsider clerics still have power. Something is answering those clerics' prayers.
Eberron contains quite a few nasty devices used by the more malicious/insane antagonists:
The Husk of Infinite Worlds, a magical device used by the daelkyr to create new species of creatures by horribly mutating others. Think of it like a DNA washing machine: creature goes in, gets put on the spin cycle, and if it doesn't immediately dissolve into primordial ooze it pops out in a new, horribly mutated state. Even then, there's only a 1% chance that the new form will be able to survive for longer than fifteen seconds. And if it does survive, the daelkyr in charge will likely just put it in again to see if it can mutate into a more interesting form.
Everything the daelkyr do qualifies as Nightmare Fuel. These are people whose main slave race was created by fusing two goblins together. These are people who bred a variety of abominations to use as clothing, including a suit of armour that's actually a limbless crab. These are the inventors of the beholder — and frankly, anyone who could develop a creature whose reproduction consists of vomiting up your young and then biting off your uterus presumably runs on Nightmare Fuel. (Important memo to whoever at TSR or Wizards Of The Coast who came up with that one: you are one sick, sick puppy).
The daelkyr also created the mind flayers. So they made a species that reproduces by putting a tadpole in your ear so it can eat your brain, take its place, hijack the rest of your nervous system, and turn your head into a squid!
The beholder was a bit of an evolutionary thing. You start with a monster that's basically a sphere that's mostly mouth and eyes, and eventually somebody asks where little beholders come from, and at that point the thing really has only one body cavity, which is the inside of its mouth. You are now imagining eyeballs erupting into tiny little baby beholders.
The Lords of Dust have devices that can turn any creature into a willing slave by pumping them full of magical sand. Here's how the process works: the subject is placed into the device — which resembles a sarcophagus — which is then sealed shut. Suckers emerge from the inside of the device, attach themselves to the victim, then proceed to suck out their soul. Once the body is completely drained, those same suckers pump the empty husk full of magical sand that turns the body into a willing servant of the Lords of Dust. Finally, the victim has a key installed in their eye socket that, when turned, will stir up the sand and grant increased strength and speed. The passage mentions that the Lords of Dust particularly love to inflict this fate on their worst enemies. Looks like whomever put these things into Eberron canon liked to watch and/or readHellboy◊, doesn't it?
One question posed to the setting creator was "Why do daelkyr look so human?" His response was "The question should be 'Why do humans look so much like daelkyr?'"
The Menace Manual has a monster called the Star Doppelganger. An alien with Voluntary Shapeshifting? Okay. Its picture is scary for some, no doubt, but likely to be goofy as hell for others. But then you read its description: an Antarctic outpost is found destroyed, the only survivor is a single dog, which gets on board the rescue ship an-holy shit I recognize this monster!
Swords and Sorcery: The Scarred Lands — The titans, humanoid eldritch abominations who once ruled over the land of Scarn with an iron fist. Despite being part of the very being of the world itself, the titans are amoral and cruel gods who will summon storms or create hideous monsters whenever they get bored. For example, Gaurak the Glutton is an obese titan who once ate all life on the once verdant moon. Their children, the gods, who ended up overthrowing them (with a few exceptions) aren't much better. Belsameth, one of the gods who controls the underworld, once got angry at a band of thieves who killed a man held for ransom, dumped him in a river, and prayed to the chaos-god Enkili instead of her to get rid of the corpse. Belsameth then bought the corpse to life, which then proceeded to murder the thieves and (even to her surprise) kill his own family for not paying up the ransom that could have saved him.
Despite defeating the titans, the gods could never truly kill them. Instead they hacked and sawed the titans into pieces and displaced them so they would never reunite. For example, the Titan Gormoth was split from skull to crotch by the gods Vangal and Chardun and each half placed on opposite sides of a great chasm. As a result, the flailing halves forever struggle to reunite.
As horrible as the Titans and most of the gods are, their created races and minions also deserve a mention in the high-octane nightmare fuel category, courtesy of Swords and Sorcery: Creature Collection:
Sword Golem - A golem made of shattered and broken swords.
Blood Horror Demon - Chosen priests who displeased the war god, Vangal, and were visited by a horrible disease that bled them until their corpses were husks. When their souls are greeted in the Abyss, things only get worse, as their souls are crafted into Blood Horrors, amorphous blobs of blood with giant maws. When the Blood Horror strikes, it prefers to leave its victims suffering from stigma wounds that, like the blood horror's death wounds in its former life, cannot be stanched.
Blade Demon - A creation of the dark god, Vangal. Among his most vicious, the blade demon appears as a twisted humanoid with scythes for hands and bladed wings wrapped with human skin. Black iron visors cover their faces, glowing with the heat of fleshly forged metal, which perpetually sears itself into the flesh of the demon.
Dark Womb - Disgusting, hideous creatures that appear as obese hags from the waist up with a huge, translucent, bulbous sac of a termite queen for an abdomen. Cursed by Mormo, the patron of all witches and hags, for trying to create life, the Dark Womb uses any tissue from a living creature to create clone children that serve her to their deaths.
Blood Maidens - Once benevolent aquatic beings that become tainted by the spilled blood of the Titan, Kadum. Blood Maidens look like beautiful women with long black hair and pale skin. When seeing their faces, the creature's monstrous nature becomes clear. Instead of a normal face, the blood maiden has only a giant circular, eyeless maw, like a lamprey.
Garabrud, the Obsidian Hound - A thing of dread across the Scarred Lands. Nigh-invulnerable, guardian mastiffs that serve the ravenous Titan, Gaurok, the Obsidian Hound will flawlessly track any designated prey, no matter how far or hidden it is. The hound will continue to close on its prey, never resting and scarcely deviating from the most direct path. For those victims fortunate enough to be far away from the Garabrud when it begins its pursuit, the hound's coming is presaged in the victim's nightmares. When a wizard tried to take refuge in a citadel to escape the hound, the intensity of the nightmares drew him to commit suicide over the year the hound spent burrowing at the bedrock of the citadel to reach him.
Hornsaw Unicorn - Imagine a unicorn. Now imagine a unicorn with a taste for flesh, pointy teeth, and a serrated horn like a rusted knife.
Skin Devils - Terrifying creatures that in their natural state appear as human beings whose skin has been expertly removed, revealing glistening muscle and pulsing veins from head to toe. Its constant agony means it must cover its form in another's stolen skin.
Flesh Strippers - Horrid packs of rodents that strip the flesh from grazing animals. The animals, anesthetized by the flesh strippers' poison, continue to graze or sleep until the beasts consume some vital organ and kill their prey.
It really says something about a setting when the city ruled by (admittedly largely non-evil) necromancers is one of the better places to live. Oh sure, there's Nightmare Fuel aplenty in Hollowfaust, but if you're a resident the skeleton street cleaners and whatnot are on your side.
The fact that up until 200 years ago the campaign setting was ruled by completely unkillable monsters who would torture or kill people by the thousands because they were bored. Hrinruuk is especially frightening (and evil). Imagine a seemingly nice guy walks into your town, gets friendly with the locals, and then demands the lord's daughter (or something else he wants) or he'll wipe out the entire city. And maybe he'll do it anyway, just for kicks, and sic a bunch of monsters on whoever survives to see how deadly they are. Then he'll kill the monsters. The worst part? He is considered the most likely to have created humans.
Dungeons & Dragons gives us the mimic, which can disguise itself as anything stone or wood. Traditionally depicted as chests, they can also appear as doors or stonework. Touch them, then you stick, and they beat you to death with their tentacles. Metal Mimics can imitate metal as well, illuminate itself, and disguise itself as a valuable artifact. Thankfully, their "eyes" are vulnerable to sunlight. House Hunters don't have the sunlight vulnerability, and can disguise themselves as a building, including light and noises. Several usually work together to form a "village". And then2nd Ed rolls around...
Even worse in 4th edition. Item-mimics are the juvenile stage. Adults? Pretend to be people.
The Ooze is a gelatinous Blob Monster whose main method of attack is to digest you alive. The 3.5 Monster Manual has a particularly demonstrative picture of a poor chap who fell victim to the monster's grasp...
The d20 sourcebook Dragonmech. A D&D type world where the moon has been moved closer and almost all life lives either deep underground or within truly Humongous Mecha , the City-Mechs up to 2000 feet tall, powered by magic, steam, clockwork... slavery, or necromancy. One of the undead 'mechs' is described as being thousands of bodies melded together... but some of the smaller ones are creepier. A meatrack is a metal skeleton with sharpened fingers as its weapons, that is powered by bare muscles attached to the frame. But even the undead ones aren't the worst... one of the character classes is a steampunk cyborg... that modified himself. Break a leg, why wait for it to heal? Hack it off and replace it with some metal... well, now you're lopsided, so you might as well do the other side too... Man, this would be easier if I could change out my hands, why not replace the arms too? Until you're left with a head on a robot body... and the steampunk cyborg did all of the work himself, aside from the implantation of the original small steam engine, was forgotten. The amputation of limbs and replacement with metal, with no mention of any sort of anaesthetic.
The self-mutilation mentioned above was added as an Artificer feat in 3.5 Eberron, aptly named "Self-forged". As in, a Warforged that forged itself. You keep your original race for a while, but eventually your self-modification will cause your character to become almost entirely Magitek.
The Derro were always a race of batshit insane lunatics, but 4th Edition has brought their insanity, and their scariness, Up to Eleven. Pale, blank-eyed dwarf-like creatures that always cackle and drool no matter what they do, their link to the Far Realm allows them to "warp" their slaves into tentacled monstrosities who barely retain any of their past features or memories. So malicious and mad are they that even the Drow will drop everything and work together to fight the Derro, because they know that if they even so much as let them get a foothold, it could mean the end for them.
We all know the stars, right? Big balls of light that you see in the sky? Well, in 4th Edition, quite a few of these stars are actually alive, and they are not our friends. These living stars are pretty much Eldritch Abominations that have one goal: annihilate the World and everything on it. Some of them are capable of creating humanoid avatars of their powers. These "Star Spawn" are all horrific in their own right.
The Herald of Hadar is a corpse-like figure that falls into the Uncanny Valley, with red orbs for eyes and wicked-fast claws. Worse, all he does is eat, eat and eat. Represented by a fat monster? Nope. The herald of Hadar is clearly starving.
Like the Herald of Hadar, the Scion of Gibbeth's sole purpose is to eat, but that's not what makes it horrifying. The picture we're given of it shows a pitch black demonic figure encased in an amorphous, corpulent shell, but that's still not what's horrifying about it. No, what is horrible about it is that this disgusting, dual-skinned form isn't the Scion's true form at all. The Scion of Gibbeth is so alien, so mind shatteringly disgusting and horrific, that no one who sees one ever sees the shape another person would see. Makes Pale Night look like a loving, friendly mother.
The Maw of Acamar is a giant, human-shaped void that leads into the depths of space, you can clearly see stars, and whole galaxies in the tear. Oh, and what is Acamar? It's implied to be a sentient, evil, black hole. One day, it might eat the world.
The Spawn of Ulban is a humanoid figure covered in purple chitin. Instead of legs, he's got seven tentacles (still purple). His hands are covered in cold, blue fire. He's more powerful than some gods, and what does he do? He destroys small kingdoms. And no, he isn't a One-Man Army. His sheer psychic presence makes people go crazy. And the worst part is that its master, Ulban, is not himself evil: he's trying to prevent The End of the World as We Know It from happening. But if Ulban isn't evil, then why are his Spawn the way they are? And why can he not hold them back?
So your players are engaged in a diplomatic endeavor. They are very lucky, 'cause they have one of their nation's best diplomats on their side. With Errol Flynnish good looks, he's the life of the party, and all the while he steers the debates in your favor. But something seems off, he has a weird accent, or odd syntax, or he steps just a little into Uncanny Valley. The PC's investigate. It turns out that this guy just popped up recently, and just in time, as a mysterious plague is sweeping the nation, and the cure lies in the hands of enemies. So this guy comes in to save the day. And then the PC's learn the awful truth: this "friend" is actually a Defector from Decadence, and his enemies are hunting him down. Won't the PCs please protect him? The plague intensifies, and kills a friendly diplomat. The other nation is struck, and refuses the to share the cure. The two nations go to war, while the dead mount. A whole continent is dragged into despair, and as the heroes look on, helpless, a purple star rises in the sky. What happened? Well, the nice diplomat was an alien monster all along. That's an Emissary of Caiphon, one of the more powerful of the star spawn. And Caiphon is revealed in a Dragon article to be the nicest of these things. He is implied to not hate you, he is just curious as to what happens when you remove part of the social foundation. Sometimes he eradicates horrors like, say, slavery, and other times he eradicates entire sentient species.
And then there is Allabar, Opener of the Way, the first 4th Edition living star we get a close look at. Remember what Atropus looked like? Well, instead of a face, imagine dozens upon dozens of unblinking eyes, as well as hundreds of rope-like "growths" around its "body." Think the moon, when it's nice and big and clear, so you can see all of the faultlines, valleys and craters. Now imagine every faultline and valley is a huge, thrashing tentacle, and every crater, from the biggest to the smallest, is a never-blinking eye. Imagine that floating in the sky above you at night. Staring at you. Hating you.
Quippers. Imagine really big piranhas that survive in cold water. Now imagine: you're walking by a pool, and then a giant fish with sharp teeth jumps out and devours you in one big gulp. And they can be anywhere. Paranoia Fuel, anyone?
Pale Night, even more than the other obyriths. Her shrouded form only exists because reality CANNOT allow what she really looks like to exist in the world. If you look under the shroud and make the save for whether or not you understand what you see, you get deleted from existence for grasping the cosmic horror you have just seen.