This is based on opinion. Please don't list it on a work's trope example list.
Nightmare Fuel / Babylon 5
Hyperspace: Everything is a bright swirling mess, currents pushing ships this way and that, if you lose your navigation systems or your engines, you will drift off the plotted route and will, in all likelihood, be lost forever. Even in the event of full-scale war, very few of the races are willing to fight in Hyperspace for this reason.
To make it worse, there are even rumors that there is... something... living in Hyperspace. The rumors are true, although the Shadows are more haunting it than anything. The actual things we see living in Hyperspace, the giant jellyfish things in Crusade, are benign, if silly.
To be exact, it's a Thirdspace alien race, who believes that they are the only race worthy to live, and is so powerful, that it took Vorlons and Shadows and all other First Ones to unite just to survive and drive them back. Vorlons explicitly stated that it destroyed thousands of species in its universe... and continues to do that. They also look like that .
Dr. Franklin gives us a vivid first-hand story about how he got to watch a friend of his accidentally get spaced. Struggling, desperately flailing his arms and legs, trying to breathe... and death by spacing is not a swift experience.
Franklin: "A lot of people make jokes about 'spacing' somebody. About 'shoving somebody out an airlock'....I don't think it's funny."
Death by Mind Rape. And if in your dying moments you should think of your friends or loved ones, they will be put in danger as well.
The Na'kaleen Feeder from "Grail" combines this with the Soldier of Darkness. The Centauri quarantined an entire planet just to contain the threat this entity posed, and they're one of the more advanced races of the B5-verse.
The entire concept of the Minbari War: one man's tragic mistake nearly dooms his entire race to extinction.
The fanaticism a species would have to condone such an action is scary too.
The Psi Corps, anybody?
Speaking of the Psi Corps, let's talk about Bester. They say that if a telepath is in contact with a person's mind as they die, part of their soul is taken as the dying person passes on. They say that Bester used to volunteer for these assignments, and that's why he is the way he is. Who would have thought the guy most famous for mispronouncing "nuclear vessels" would go on to play one of the most sinister characters in science fiction?
Talia Winters: "Psi Cops are trained to make people feel uncomfortable. Bester makes other Psi Cops feel uncomfortable!"
And then there's the alternative to the Psi Corps—being administered sleepers on a regular basis, which over time dull the mind until there's little (if any) personality left.
Ivanova's mother was made to do that. She killed herself.
Throw in the mentality that CREATED Psi Corps. Ironically it's Garibaldi who points it out (sort of giving a What the Hell, Hero? speech to humanity: "We put them all in a black box labeled 'Psi Corps' and now look, black uniforms, jackboots..." He openly points out people did it because they were scared of telepaths. In the Psi Corps novels it's estimated that when telepathy was proven real, thousands of telepaths were murdered (and the books set before the series begins show that there were still plenty of people who hated telepaths enough to kill them, or lure would-be rogues into horrible fates like sex slavery.) Several characters explicitly compare the telepath crisis to the Holocaust (and on the show Edgars refers to "the telepath problem", as in the "Jewish problem", when discussing his own...final solution. Bester's hardcore determination to put teeps' safety over mundanes (even crazy teeps or rogues) makes a lot of sense when you consider that not only was one of the richest humans in the galaxy willing to create a virus that could exterminate telepaths, but he found a lot of people willing to knowingly help him.
Pretty much the entire episode "Intersections in Realtime", partially because it's done in a way that feels very brutal and real.
The ending of the episode is particularly brutal. Sheridan's spent the entire episode being tortured by Clark's regime, his torturer a bureaucrat who tries to make Sheridan confess that alien influence compelled his rebellion. After being tortured viciously and continually refusing to break and sign the confession, his torturer urgently presses that it's last chance before he's sent to Room 17 to be executed. Sheridan refuses and is wheeled to Room 17. Where he's secured to a chair and a completely new bureaucratic torturer comes into the room to begin the cycle of torture all over again, repeating the exact same opening speech the first torturer said verbatim.
JMS used notes from Amnesty International and PEN International, advocates for political prisoners and real interrogation techniques.
As awesome as it was, Lyta mind-controlling the entire bar in "Wheel of Fire". Just imagine that you're one of those customers.
G'Kar's whole experience with Dust. From his rage-distorted features, to him flinging Vir around like a ragdoll (it's not even clear if Vir is alive at that point), to his Mind Rape of Londo, where he carried a disturbing resemblance to Freddy Krueger.
Wasn't there an episode where Garibaldi had to confront a psycho dummy? If that were the case, then that was likely one of the most scariest episodes.
The Drakh and their Keepers. Particularly in the Centauri Trilogy books, Shiv'kala (the lead Drakh operative on Centauri Prime) is described as materializing "out of the shadows" in a room, and blending back into them when he's done meeting with/confronting Londo. Plus the intensity of pain the Drakh can transmit through the Keeper onto the one who bears it is described in a bit more detail in the books. The whole concept of being controlled by such a thing for the remainder of life may be something described as a fate worse than death.
The Passing of the Techno-mages trilogy had its share of nightmare fuel, particularly in the final novel with the things Galen saw on Z'ha'dum in its industrial hub surrounding the Eye: masses of live sentients herded down grim corridors to be dismembered and used to make the "substrate" of Shadow vessels, or their neural systems being made into Technomage chrysalises. As horrible an imagining of hell as anything, in those vast underground corridors. Then imagine Weirden (who Galen was intended to replace) being forced to be the neural hub of the Eye, which directed all that activity, for 1000 years.
The various scenes of the "Shadow surgeons" and their whirring instruments (all depicted from the vantage point of their victims lying prone beneath them), to Carolyn's flashbacks ("Ship of Tears"), to Londo's being prepped for his Keeper. All of these are remembered as nightmares, implying their Playing with Syringes is done while the victim is in some kind of sleep or trance but the mind remembers it.
Mr. Morden: "Flesh does what it's told." Brrr.
Emperor Cartagia: scarier than the Vorlons and the Shadows because he's all too real. The nightmare of dealing with a madman - of never knowing whether or not what you say or do will set him off and cause him to kill you and your entire family. Also? He just sold out your planet to a race of eldritch abominations. Sleep tight!
The worst part? He wasn't tricked, or lied to, at the most the "cost" of the deal was underexaggerated. This man pulled a Deal with the Devil on behalf of his entire race, willingly and knowingly. And he's the one in charge of the second or third most advanced of the "young" races.
And that as pure and unsullied a spirit as young Vir had to commit murder to stop him.
The very concept of Soul Hunters, in a universe where souls are not only real but physically observable. Imagine being denied whatever afterlife may exist/you personally believe in just to be confined to a jar for someone's "collection" for all eternity. No wonder most races react to the Soul Hunters with extreme hostility!
In The River Of Souls the soul hunter visiting the station displays an orb that may have the souls of the men who robbed the Soul Hunters. And those souls are screaming in agony.
Even more frightening. . . what if the Soul Hunters are right? What if there is no afterlife, and the soul ends with death, and the only way everything you are is not lost forever is to be placed in that collection?
And then. There's the fear that the Soul Hunters invoke with just their presence. They like to pop up when they know someone important is about to die. As in these bastards instinctively know when someone's gonna die, even if that person has no idea. As individuals, they aren't any less vulnerable than the average intelligent species. However, if one sees their ships hanging around, it isn't because they really want just the one soul; they're there to mop up all the loose souls once the massacre is over. No one takes notice of Soul Hunter ships once the killing begins, and, when it's all over, there's no one around to stop them.
Catherine Sakai found one of the "First Ones" by accident at Sigma 957 and almost didn't survive. Now imagine all the over expeditions that didn't make it. Sure you might find mineral riches or some valuable artifact or you find Sealed Evil in a Can. Pray you never run into some of the First Ones. Some are just indifferent others are downright sadistic.
The interrogator from Intersections In Real Time. He looks about as menacing as an accountant, which only makes the things he does to Sheridan worse. It takes Cold-Blooded Torture to a new level, as he treats the whole thing as being as boring and routine as doing your taxes. Sebastian from "Comes the Inquistor" at least had the decency to look the part of a sadistic torturer.
The mass driver drops asteroids on the target planet. It was so bad that the Centauri (who invented it) made treaties to ban its use against planets after the war they invented it for ended with numerous worlds made uninhabitable as collateral damage-and when Refa broke the treaty even other Centauri are disgusted, even if it was being used against the wholly hated Narn.
The Shadow Planet Killer, also known as the Death Cloud: an immense ship made of smaller components that fire missiles that burrow in the target world's surface until they reach the mantle, at which point they explode with enough strength to shatter the surface. Again, they have at least two of these.
The antimatter converter, whose beam turns the target's surface into antimatter and lets it react to the non-converted matter in contact with it. A single standard-sized ship armed with this can lay waste to a planet in hours. Two races have it: the Vree, who reverse-engineered it from relics, and the Minbari, who invented it before or during the last Shadow War. That's right: the Minbari invented a device that's possibly more horrifying than the Shadow Planet Killer when they were far less advanced than they're now.