This is based on opinion. Please don't list it on a work's trope example list.
Nightmare Fuel / Star Trek: Deep Space Nine
The Battle of Wolf 359. Ships swoop in firing at the Borg cube. And then the cube fires back, wrecking all of them. The Saratoga gets blow to pieces. Sure Jake was knocked unconscious, but what about the other civilians? The ship then explodes as Cmdr. Sisko watches from his escape pod. And the Borg just keep moving....
In "If Wishes Were Horses", one of Kira's imaginations-made-real has her encounter a corridor in flames, and a man on fire screaming running right at her and make her share his grisly fate before disappearing at the last second.
"...Nor the Battle to the Strong" has Jake knocked out by Klingon shelling. He wakes up on the remnant of a battle, surrounded by bodies of Klingons and Starfleet personnel. At one point, you can see a bat'leth sticking nearly vertical out of a Starfleet corpse.
Cardassian atrocities during the Bajoran occupation. In "Duet," Kira describes the horrific conditions at the Gallitep labor camp she helped liberate.
Kira: All those Bajoran bodies, starved, brutalized. You know what Cardassian policy was? Oh, I'm not even talking about the murder. Murder was just the end of the fun for them. First came the humiliation! Mothers raped in front of their children, husbands beaten until their wives couldn't recognize them, old people buried alive because they couldn't work anymore!
The Cardassian attitude towards the Bajorans, chillingly summed up in this monologue:
Darhe'el (actually Marritza): Oh, no, no, Major, you can't dismiss me that easily. I did what had to be done. My men understood that, and that's why they loved me. I would order them to go out and kill Bajoran scum, and they'd do it! They'd murder them! They'd come back covered in blood, but they felt clean! Now why did they feel that way, Major? Because they were clean!
In The Jem'Hadar, the titular aliens attack a retreating Galaxy-class starship (the same class as the Enterprise from TNG). And the Jem'Hadar destroy it by ramming it. All to send a message to the Federation, the war goes downhill from there.
O'Brien: "We were retreating. There was no need for a suicide run."
Sisko: "They're showing us how far they're willing to go."
And it's not just about sending a message either. As we learn later, Jem'Hadar and Vorta are extremely easy to replace and the founders do not regard their lives as valuable. This means they can afford to think of strategy like they're playing a game all the time. As long as the other guy loses more than they do, the engagement is a success.
Illnesses and severe injuries to Changeling characters play up the body horror trope. For example, in "Broken Link," Odo has difficulty maintaining solid form due to an infection inflicted on him by the Founders. It's... not pretty.
The morphogenic virus created by Section 31 deals a serious dose of body horror to Odo and the Female Changeling in the latter episodes of season 7. Their physical deterioration is visually disturbing.
And made even more disturbing when you compare the deterioration with that of smallpox (thankfully an eradicated disease) and leprosy (that is not eradicated, and still difficult to treat).
When Garak tortures Odo using a quantum stasis device in "The Die Is Cast," Odo is unable to regenerate, and his appearance becomes horrific.
Let's not forget Mirror!Odo's messy explosion when Bashir shoots him with a phaser in "Crossover."
Lovingly called "Explodo!" by the fans.
"Second Skin," a solid hour of Gaslighting. Kira wakes up as a Cardassian - yes, Kira, who fought in the Bajoran resistance for years - and is told, for days, that her whole life as a Bajoran has been a lie and that she was born Cardassian, with her captor continuously giving her stories and evidence until she reaches a Despair Event Horizon. So messed up.
In "Facets", Dax interacts with each of her previous hosts in the person of her friends. Most of these are cute, sweet or funny, but then we get to Joran. Not only does Avery Brooks use an incredibly creepy voice to Hannibal Lecture Jadzia, he uses the security forcefield to Inflict Electric Torture on Sisko. When Jadzia drops the field, Joran!Sisko attempts to strangle Jadzia with one hand.
Apparently, there's a take of Joran that's even creepier - because Avery Brooks spoke his lines in a nigh-inaudible whisper which really creeped the directors out, forcing them to re-shoot the scene.
'Hard Time' brings us the Argrathi 'prison system', which uses implanted memories to compress years of horrific confinement into a matter of minutes. O'Brien gets a full two decades in a couple of hours after being charged with "espionage" (read: asking polite, innocuous technical questions), and the impact it has on him is both heartbreaking and utterly terrifying.
Another 'O'Brien must suffer' episode, 'The Assignment', brings us the very memorable first appearance of the Pah-Wraiths. One of them possesses Keiko and uses her as a hostage in an efficient and sadistic manner that's all the more unsettling for how casual it is. The sequence where the Wraith calls O'Brien about his rapidly-approaching deadline whilst combing Molly's hair is pure Adult Fear.
The Pah-Wraiths in general, in fact, are just plain creepy, as are their followers. "Covenant" throws Dukat's creepy personality into the mix to make him an all-too-believable Jim-Jones-style cult leader. What's most nightmarish of all about this cult is that they're not kidding and neither is Dukat (though he's not being straight with them about his actual motives for serving the Pah-Wraiths either). As Kira states at the end of the episode, he really does believe in his new-found religion, which just makes him all the more dangerous.
Number three for O'Brien's suffering on the page, we have Whispers. Long story short, the O'Brien narrating the episode is actually a clone, who is killed at the last minute of the episode. The terror, as pointed out by SF Debris, is that no one realizes just how perfect a clone he was. Except for the mental command to attack a senator, the clone!O'Brien is in no way different from the real one. Same memories, same emotions, and with no evil tendencies. He's just... killed. Unceremoniously. No one in the room cares that he's dying. He did absolutely nothing to deserve his fate. In fact, his efforts to prevent the attack (he didn't realize that he was the one who was supposed to carry it out) are extremely heroic. Dying scared, confused, and alone.
The Breen. They only appear in a handful of episodes, but they certainly make an impact. For one, their technology is different, more advanced, than anything the Alliance has, able to destroy the Defiant in five minutes. Second, we know NOTHING about them; their culture, motives, even what they look like under those suits, all are left mysterious. And finally, we don't even know what they're saying. The Universal Translator seems to translate their speech, but it's all R2-D2 Speak to the audience. All we take from them is what we see: they're clever, unpredictable killing machines with absolutely zero mercy. The Breen epitomizeNothing Is Scarier.
The first Breen contribution to the Dominion's war effort is to launch a successful raid on Earth—the heart of the Federation, and a planet so heavily fortified that the Klingons never even considered attacking it.
In "The Quickening", Bashir thinks he's cured a blight that has affected a planet for centuries. Needless to say, he realises he hasn't when people start screaming.
But he DID cure it in a sense. While the cure didn't work on the pregnant woman, it DID cure her unborn child. So by using his treatment, the next generation would be free of the disease. This is pointed out to Bashir repeatedly, but he still feels guilty that he wasn't able to cure the current generation and promised to keep working on it.
In an episode in which someone is killing Kira's resistance cellmates, they send her messages saying "That's one", etc. as each person is killed, in a deeply creepy Saw-type distorted voice. Even creepier? They use Kira's voice.
There is a dose of Body Horror when we find out who is the killer. It gets worse when we find out that he's like that because of Kira. And there's an uncomfortable dose of Fridge Horror as we find out that she regularly crippled and mutilated Cardassians incidentally as part of her resistance activities. The fact that she found all Cardassians on Bajor - including children and innocent bystanders who never picked up a phaser - legitimate targets just adds to the Nightmare Fuel. Turns out Bajorans and Cardassians aren't that different after all, and not in a good way.
In "Field of Fire", when the insane Vulcan murderer is revealed, he says he committed his crimes because it was logical.
Another creepy moment happens when Ezri and the villain, both using guns which can see through walls, find themselves aiming at each other, at the exact same time, from all the way across the freaking station.
Also, their weapons, the TR-116: guns that can see through walls, have integral silencers, and, with a small modification, can teleport bullets.
And if you think it's bad, let's add Fridge Logic: as the miniature transporter attachment isn't readily apparent, O'Brien was apparently aware of it, and the sensors could be useful only for that, there are strong hints it was intended to act with the transporter. Then, let's examine another feature of the gun: it used large 9-10mm subsonic rounds, too slow and short-ranged for combat use without the transporter attachment (that could be easily jammed in combat condition) and large enough that the magazine could only hold 2-3 of them at a time. What can you use a short-ranged sniper rifle with pistol-caliber teleporting ammunition and a small magazine whose transporter can be easily jammed? Hostage situations... Or assassinations.And it was built by the Federation. Let's just hope the transported attachmentwas an addition to the design and the sensors that can see through the walls were just added by a designer showing off...
Serial killing aside, the targeting sensor alone introduces another level of Fridge Horror: it allows the user to look into someone else's private quarters without that person's consent, or even knowledge. Power Perversion Potential, anyone?
And the reveal that the Founders weren't even directly responsible for the damage done in that episode.
That lovely scene when the Bajoran woman hangs herself on the crowded Promenade.
It really hits home in the almost anachronistic method of it all. Vedek Yasim puts a rope around her neck, ties it to the balcony, and lets herself fall. In a show where the deaths are usually caused by phasers and disrupters, the way that she kills herself in the same way that someone could do so today is all the more jarring.
The Dominion has "Houdini" anti-personnel mines, which hide in subspace and make you "disappear" — at a randomly chosen instance, not by predictable rules. So basically, if you have no way to detect them, nowhere is safe anywhere they've been laid, even places you've passed hundreds of times.
There's Fridge Horror in this, too. Houdinis were poor anti-personnel devices, and absurdly bad area-denial weapons; they exist only as weapons of terror—designed to torture the Starfleet personnel with constant stress and paranoia.
What makes the most horrifying scene of all involving them so awful is that the horror is retroactive; you may just feel the need to recite an expletive when Sisko & company finally get their device working to strip away the mines' concealment, and you get to see just how many of them there were. Not even counting the ones that had already detonated.
"Far Beyond The Stars" is just full of terrifying concepts. The denial of freedom, social ghettos, and sickening racial prejudice accompanied by dehumanization being shown as reasonable common everyday occurrences is bad enough, but Ben screaming "I'm a human being!" and ranting in despairing protest is highly disturbing in more ways than one. That it really brings home the horrifying reality of being stuck in a world where you are treated as subhuman and inferior just because of your skin color (or gender... or any other distinguishing genetic feature/s which manifests in a phenotype that contributes to appearance) is just an added bonus, of course. The accompanying Fridge Horror that even nowadays racial tensions in America are still a serious social undercurrent is just not helpful.
Dukat's obsession with Kira gets creepier and creepier. The scene in the captain's office during 'A Time To Stand,' where Dukat places his hand on Kira's cheek can easily be seen as a prelude to him attempting to rape her. Remembering that to the Bajorans, Dukat is essentially Hitler makes it unsettling, even before factoring in his relationship with her mother.
Tribunal, being another "O'Brien must suffer" episode has the scene where, after being taken prisoner by the Cardassians, O'Brien is "processed", for a crime he has supposedly committed (except no one will tell him what that crime is). He is ordered to remove his clothes and when he refuses, instead stating his name, and rank, and asserting he is a Federation citizen, he is forced against the wall by two guards who proceed to forcibly rip his clothes off him. He is then forced onto the floor where a retinal scan is taken (and from O'Brien's reaction, it's suggested the scan is painful) before he is dragged to a rather sinister-looking operating table. He briefly tries to escape but is punched in the face by a guard and restrained. Then he appears to be drugged with some sort of hypospray, while a lock of his hair is cut off and a tooth is forcibly removed, without the use of anesthetic. It's even more graphic and unpleasant to watch than what Picard went through with Gul Madred in TNG, and this is before we even get to the mockery of justice that is Cardassian jurisprudence. Earlier in the season, during The Maquis two-parter, Dukat had arrogantly boasted about the "perfection" of the Cardassian legal system (that the verdict is always decided beforehand, it's always guilty, and that "trials" are really nothing more than scripted televised farces designed to show the terrifying power of the Central Command, and also, it's implied, as a twisted form of entertainment for the public who enjoy seeing "justice triumph over evil") except now we actually get to see how terrifying this system actually is.
It's only made worse by O'Brien's counsel trying oh so hard to make him throw himself on the mercy of the court. Hearing a proud fascist guilt-tripping an innocent man into confessing to a crime he didn't commit "for the sake of the children" — one of the favorite justifications for real-world fascists — sends chills down the spine.
It also makes the plot of Duet make sense in a rather sickening way- Maritza thought that he would have a Cardassian style trial, where he would be told "You're guilty, send him to the execution!" After pointing out all of his crimes, of course, which would expose the cruelties of Cardassia to the Alpha and Beta Quadrants. In a way, it makes it far more heartbreaking what actually happened- he expected he'd only need to keep up the charade for a short time, and his already broken mind shattered completely once the ruse was exposed. He went there to die, to be free of the nightmares... but they didn't play according to the rulebook he knew...
Q's simulation/reenactment in Q-Less of what would have happened to Vash from her insect bite if he hadn't cured her. Seems unusually cruel for Q (at least to humanoids), even if he was about to cure her again.
When the team go to save Kira in "Second Skin," Entek tries to shoot them in the back while they're leaving. Quick as a shot Garak fires first, disintegrating him, then casually comments "Pity; I rather liked him," before hurrying off, leaving Kira and Odo to share an aghast look.
Throughout the series, whenever Garak kills someone, he always does it with a pleasant, but deeply unsettling smile on his face.
For that matter, his willingness to kill (or indeed, allow people to die) to resolve a situation always horrifies the people around him, even those (like Kira ) who are used to death and using killing to attain a goal.
Cranked Up to Eleven in "Empok Nor", when you find out just how terrifying Garak really is once you take away the smiling façade. The whole episode is a horror movie in space.
More subtle than Empok Nor, and all the more terrifying for it, is Garak's handling of Tolar, the holo-forger from "In the Pale Moonlight." The episode makes much of Garak's assassination of Vreenak, a Romulan senator, but for that Garak required Sisco's full, albeit unwitting, cooperation. In contrast, Garak "disappeared" Tolar, off-screen and without attracting attention from ANYONE, not even Odo, over the course of two days. It's only realized he did so when Sisco makes the disturbed inference that this side of Garak would never allow dangerous witnesses to live past their usefulness.
In its way, the look on Kira's face as she walks down the corridor to Odo's quarters at the end of "Behind The Lines," after Odo failed to deactivate the security monitors leading to Rom's arrest by the Dominion. There is a look of absolute fury and murder in her eyes, and she POUNDS on the door panel to his quarters to demand admittance. Considering that Odo is one of her best friends, the fact that this is directed towards him specifically...
"Business as Usual" at first tries to downplay Quark's role in selling weapons, until his conscience catches up to him in a bad way when a client wants a weapon that can kill 28 million people. He has a nightmare in which the animated corpses of the station staff, including O'Brien holding his dead infant son, accuse him of murdering them. The stress and guilt cause him to take an action which risked his own life just to put an end to this.