The aliens who gave him his powers and apparently raised him. They're apparently so creepy that Charlie himself is frightened of them. You know, the same reality warper who just spent the whole episode swaggering about invincibly, smugly confident in his own superiority, and now he's begging the same people he was just bullying not to let them take him away. Then the way his last plea of "I wanna stay" echoes when they teleport him off the ship...
"Spectre of the Gun": Kirk and the gang are trapped in a nightmarishly surreal world where the clock is ticking to an unavoidable execution and there is seemingly no escape to that conclusion.
There's the classic "The Doomsday Machine", where a giant, bluish-green horn of plenty like machine is consuming everything in its path and the mighty USS Enterprise can't seem to stop it. Not to mention its massacre of the crew of another ship, identical to the Enterprise, whose captain the experience had driven completely mad.
The machine is pretty much a preview of the Borg—unthinking, unfeeling, and feeding its insatiable hunger by devouring whole planets for unknown ends. Add to that that it is made of metal so dense that the entire Federation fleet could unload its entire arsenal on the machine and it wouldn't be phased, and is, by a wide margin, the most powerful weapon ever seen in the franchise (more-or-less on par with the Death Star) and you have one pants-shittingly scary machine. And we never find out who built it, what they built if for, or if there are more of them out there somewherenote Well, we do in the extended universe—it was actually built to take on the Borg—but it is decidedly not canon.
"The Man Trap", with the shapeshifting alien that sucked the salt out of the victim's body. And it was the very first episode to air!
That thing was literally responsible for three years of nightmares. ugh.
And the Horta: imagine a giant pizza out to kill you.
At least the Horta turned out to be nice — it turns out she was only defending her babies. The gigantic brain cells of "Operation: Annihilate!", on the other hand...
They made Spock scream. Spock. Just trying to imagine the level of pain that would require is Nightmare Fuel all by itself. And to add further weight, Spock himself thought permanent blindness was a fair trade for not having to face that pain again. Luckily for him it wasn't permanent, but still...
Moreover, remember that Kirk's nephew, not a Vulcan, is facing potentially far worse pain, should he awaken from his coma — and despite the joking around at the end, Kirk still has to tell the kid that his parentsnote and perhaps his brothers too, though we never see them died in agony, along with a sizable amount of everyone he knew.
In the episode "What Are Little Girls Made Of?" you have Ruk, played by Ted "Lurch" Cassidy. Imagine an extremely intimidating giant who is extremely strong and yet also moves with such deadly grace that he can grab you long before you can react.
Weirdly, his clothing adds to the horror. It's utterly bizarre to see a ruthless android casually murder Red Shirts while dressed in something your grandmother might wear to bed.
The nightmarish ending of the episode also counts: the Andrea-bot, wielding a laser pistol, utterly at odds with her programming - unsure of whether to love or to kill, embraces her creator, Dr. Korby's android double and Korby pulls the trigger, disintegrating them both. Kirk and Chapel look on in horror, as do we.
"A Taste of Armageddon": The people just walk into the disintegration chambers, like they're off to work.
The cloud creature in "Obsession". It's capable of space travel, phasers don't do squat against it, it can silently sneak up on its victims pretty much anywhere, and if it catches you it basically sucks out your blood without even leaving a mark.
The return of the Cloud Monster of Death was the premise of a DC Star Trek Graphic Novel written, IIRC, by Chris Claremont.
And the part where the Red Shirt of the week beamed up to the ship, and then dropped dead? And then a scary voice came out of his dead mouth? Yeek.
The Kelvans, who reduced a young yeoman to a polyhedral cube and then crushed her to dust! ("By Any Other Name")
The Zetarians in "The Lights of Zetar". Non-corporeal energy beings who zoom around the galaxy so fast the Enterprise can't outrun them, searching for someone to possess so they can live out their lives. And if they can't possess you, they'll just kill you horrifically while trying. The woman who dies on the station spends several seconds with her face writhing uncontrollably and glowing several different colors, possibly in a very great deal of pain, before she dies. Even with the long-out-of-date and obvious special effects, the shot is still unnerving.
The episode 'The Trouble with Tribbles' is mostly a lighthearted comedy, but the scene at the end where dead tribbles fall on Kirk until he's standing nearly waist-deep in them is very... disturbing. It's the equivalent of opening the door to the attic and being bombarded with dead kittens.
And the way that they just kept falling as Kirk was standing there, probably from being casually tossed aside by Dax and Sisko. And it was played for comedy. Twice!
Chekov under the effects of the Hate Plague in "Day of the Dove." Seeing the goofy Plucky Comic Relief character with a bad accent suddenly start attacking everyone and attempt to rape some poor lady while whispering creepy things to her was really... disturbing.
Balok in "The Corbomite Maneuver". Both as young Clint Howard with that adult voice and the alien dummy he pretended to be were scary. The latter even more so because they put that image at the end of every closing credits: you had to see it if you watched every episode from the second season on!
Khan Noonien Singh. Granted, he would get much worse in the second Star Trek film, but even in his introductory episode, "Space Seed", the man was frightening. During his attempt to take over the Enterprise, he makes it clear to Kirk that he isn't screwing around by locking Kirk, Spock, and Uhura in the bridge and shutting off Life Support there in order to get them and the crew to surrender. When that doesn't work, he forces them to watch each other die, one by one, via suffocation in Sick Bay's decompression chamber.
Nobody deserves the hell that Khan and his followers were forced to endure between Space Seed and Wrath of Khan. How many people were attacked by the Ceti Eel before they figured out what was happening?
Khan: It killed twenty of my people...including my beloved wife.
Almost everything about NOMAD from "The Changeling", from its genocidal mission to "sterilize imperfect beings" to sending Spock into a mind meld induced Madness Mantra to wiping Uhura's brain to killing Scotty when he tried to defend her.
And there's no Reset Button for Uhura! She has relearn everything she knows, and probably lost her entire lifetime of memories!
The Constitution-class ship is kind of bright and cheery with the red doors and uniforms and such, right? Well, when everyone on it is dead or almost dead, the emptiness is kind of creepy, creepy like an Abandoned Hospital. Specifically, the Defiant from TOS: "The Tholian Web" and ENT: "In a Mirror, Darkly" and the Republic from Star Trek: 25th Anniversary were messed up real bad. It didn't help that Kirk and his landing party on either ship were stuck there without the ability to transport to the Enterprise — that's right, no escape route despite being on ships falling apart at the seams. And in the former case, on a ship that is phasing in and out of reality as you know it.
It was also a bit bad in "The Mark of Gideon" when Kirk is stuck on an eerily empty Enterprise. Loneliness is a sort of hell, particularly for an extreme extrovert like Kirk. Oh, and hello, right before the first ad break, a bunch of pallid faces fade onto the viewscreen without warning, just staring...
And "The Tholian Web"; the idea of an entire crew gone stark raving mad to the point where they all killed each other.
The agony booth from the episode that introduced the Mirror Universe, not to mention just imagining what life must be like on a day-to-day basis in the mirror universe...
And the agonizers, the torture devices that crew members carry around on their own belts, forced to surrender them every time an officer is displeased with their performance.
The Neural Neutralizer from "Dagger of the Mind". The device was originally intended to cure the mentally psychotic, but one scientist decided to make a few..."minor adjustments". Not only does it inflict as much pain as the agony booth, but the operator can make changes to a patient's personality and memories. In the off-chance someone taps into his or her true self, they are inflicted with intense pain, as shown by poor Dr. Van Gelder. Probably the worst part is when a person is in the chamber with the device at full blast and no operator present...
Kirk: Can you imagine a mind....emptied by that thing?
Gary Mitchell from "Where No Man Has Gone Before", especially the part where Kirk and Spock are monitoring him reading through the ship's library in minutes flat and he turns and stares at them with his creepy, silver eyes. Then there's his powers, being able to make an oasis in a barren wasteland, and wanting to use said powers to make himself a god.
Not helped that those assholes (the psychics) more or less make Dr. McCoy watch the whole thing in an attempt to make him stay on the planet as their physician, not long after he saved their leader from dying.
The flamenco the Platonians forced Spock to do almost ended in him stepping on Kirk's face and possibly killing him. When they let him go, the Vulcan collapses — all his strength and willpower were expended keeping his foot those precious few inches away from Kirk's skull. Not only can the Platonians hurt you themselves, but also they can force you to hurt those you care about, completely conscious of what you're doing yet unable to stop it. That's gonna give you some major PTSD.
"Wolf In The Fold":
The idea that a seemingly immortal being had been possessing and killing people for thousands of years just to feed on their fear.
"The Immunity Syndrome": An enormous space amoeba devours the USS Intrepid, the same class vessel as the Enterprise. What must have gone through the minds of the all-Vulcan crew when they realized that their curiosity and logic had failed them? Worse—like any other life form, the space amoeba can only exist as part of an ecosystem. There are no doubt more space amoebas...and they may be at the bottom of the interstellar food chain...
"Miri": These children have been left without adult guidance for centuries. No one to calm their fears, or to show them how to survive as the infrastructure around them crumbled and decayed. No one to protect smaller children from bigger children who would exploit and abuse them. Infants starve to death as woefully-unprepared pre-adolescents attempt to care for them; these children in turn must deal with the trauma of watching their younger brothers and sisters die despite (or perhaps even because) of their best efforts. Puberty and adolescence becomes not a time of growth and new experiences, but a harbinger of death. The novelization makes it clear that the children face starvation as the preserved food they'd been scavenging from the ruins runs out. If the Enterprise had arrived just a few weeks later, the entire population of the planet would succumbed to famine. Once these children do finally grow up, they face a lifetime of psychological trauma and dislocation.
In a more subdued sense, McCoy during his injection-induced paranoia in 'The City on the Edge of Forever". He's wild, insane and irrational, and entirely unable to recognize former friends. But we also see he's very good at stealth, and drops the transporter chief with an expert attack... that's right, he retained his knowledge of physical training (not to mention basic transporter operation). And Spock's nerve pinch later in the episode only keep him down for ONE minute, where most are out for an hour or more.
"Amok Time" is horrifying from Spock's perspective. Losing your mind, losing your self-control, hurting your friends...To make it worse, Heroic Willpower and The Power of Friendship have next to no effect.