This is based on opinion. Please don't list it on a work's trope example list.
Nightmare Fuel / Brutal Doom
The Spectres. The partial invisibility is gone. Instead, they are completely invisible except for their glowing red eyes. And these things move fast, and are just as intent on nomming you as their pink cousins.
In addition to their regular fireballs, the Imps can now leap right into your face to claw you to death, just like their counterparts from Doom 3.
The Spider Mastermind and Cyberdemon are even more terrifying. Thought they were intimidating before? How about when every attack they make shakes the earth?
One of the Brutal Doom Mutator side-mods changes the zombified soldiers' grunts and snarls with voice samples, often saying things to antagonise the player. Now imagine hearing things like "You will not escape!" just when you quit the game from main menu.
Just as bad, imagine going through a place like the maze in "Nuclear Plant", E 1 M 2, and hear enemies say "I smell fear..." or "I smell human..." as you get near to them, then scream "FLESH!" or "Hell awaits you!" when you enter their line of sight. Occasionally, when active but out of line of sight, they'll mutter "Sector clear..." giving the impression that they're hunting you.
And that same mod allow imps to crawl on the ceiling to hunt you. Each growl you hear, you'll start looking up frequently.
One of the game's integral features is that when you kill an opponent with your fists, it triggers a Fatality. And even in a mod as gory as this, they still stand out for just how grotesque they are. Picking your enemies up and tearing them in two, ripping out organs, crushing skulls under boots, literally beating faces in with your fists...
Oh, and some enemies can deliver fatalities too. Against you, and against each other. Pinkies/Demons and Spectres, for example, will grab an enemy and stuff them halfway into their mouth, chewing on them and sending gore flying before swallowing them whole, while Barons of Hell will grab zombified marines, rip them in half, and then throw the bodies at you. Revenants will grab imps by the leg and pound them repeatedly on the ground, beating them to death before hurling them at you.
Let's take a look at some of the fatalities, shall we? As much as they're awesome, they're certainly unnerving.
Imp: Throwing them to the ground before crushing their skull under your boot, ripping their heart out with your bare hands, knocking them down and then punching them in the face until you crush their skull.
Cacodemons: Grabbing their upper jaw before heaving upwards to tear them vertically in half, or ripping out their single eye.
Pain Elemental: Ripping off one of their horns before throwing them onto their back and stabbing them in the eye multiple times with it as blood splatters all over you.
Mancubus: Ripping open their guts and leaving them slumped to their knees, groaning in pain as their entrails fall out.
Chaingunner: Tearing them in half, or throwing them to the ground and ripping their arms off before leaving them to bleed out. They scream for a minute before this happens.
Baron of Hell: Ripping his arm off before using the claws on his own severed hand to decapitate him, or tearing him open with your bare hands.
Hellknight: Rolling underneath him, grabbing the base of his spine and then ripping out his spinal column.
The fatalities on the player really drive home how utterly screwed you were the moment you stepped out of that fateful hangar in how humiliating they are:
Zombie Sergeant: executes you Bodicker Style, blowing your arm off and leaving you to bleed out, screaming in agony.
Baron of Hell: Absolutely dominates by ripping you in twain from the neck down to the pelvis. Slowly. Letting the blood gush out and spurt everywhere. Your screams don't last long. Even had you stayed conscious for the whole thing, you'd be wishing you weren't.
Hell Knight: Throws you to the ground before stepping on your head, bursting it like an overripe tomato. Like a boss.
Revenant: makes you his bitch by effortlessly ripping your head off, your scream cut short , lifting the separate parts in the air before dropping them with a shrug.
Mancubus & Arch Vile: Total immolation with hellfire, making you wail miserably as your body flails and convulses in excruciating pain.
Some of the levels from the Starter Pack that came with Brutal Doom v20 are downright terrifying and push the Doom engine to its limits. In particular is "Sacrificial Grounds", a bona fide demonic concentration camp which features captured marines chained up and mutilated, screaming in agony as if the're begging you to put them out of their misery. Even though a certain variation uses the Headcrab Zombie sounds from Half-Life 2 in order to make it even more disturbing. The music, carried over from Doom 64, is also punctuated with low sorrowful moans and crying.
The bosses are also much tougher to beat and scarier.
Instead of two Barons of Hell at the end of the first episode, you fight two centaur-like demons who fling green explosive BFG-like blasts at you.
The Guardian (which is Korax from Videogame/Hexen) guarding the portal between city and hell rains fire on you from above.
The giant red cyberdemon in the tower is similar to the boss at the end of episode 2, but he has two rocket launchers and can spit fire.
And the Icon of Sin, possibly the largest demon you will ever face is now an Eldritch Abomination with a full body and can one-hit kill you in addition to summoning enemies.
Sometimes a well-place hit will blow off limbs. Sometimes this is comical, like an imp hopping around uselessly on one leg. Sometimes it's a bit morbid, like enemies crawling around or otherwise incapacitated, screaming their little heads off. But it's more unsettling when that isn't enough to stop them. Revenants might be left crawling after you and still firing rockets, zombies might draw a sidearm after you render their weapon hand useless, etc. For added fun, regular demons that lose an arm will stand there stunned for a moment. They might be out of the fight, or they might continue charging after you shortly afterwards.
The Fearrific add-on for the Project Brutality expansion of Brutal Doom makes the game much scarier, with darker, more realistic textures, enemies' eyes glowing in the dark, and new music that's often downright frightening. Then there are the new sound effects, which give Imps horrifying cries and Pinky Demons footstep sounds, among other pleasant noises. But perhaps most of all, the Specters get a major upgrade: Instead of just being partly invisible Pinkies, these new Specters are ghostly imp-like creatures that are completely invisible (lacking even the glowing red eyes of the Brutal Doom Specters), with no warning other than ghostly swirls indicating their location, until it's too late and they appear to slash at your face. Whenever you encounter these guys, prepare to back yourself into a wall and frantically mash the melee key until you're sure they're all dead.