The possibility of Angelus. To elaborate, Angelus is one of the most dangerous vampires, if not the most, in the Buffy-verse, and he is always inside Angel. Those around Angel (Cordelia, Wesley, Gunn, Fred, etc.) have to live with the fact that if just a few things go wrong, they might be facing "The Scourge of Europe," whose mere voice turns Cordelia into a whimpering, shaking mess. It speaks volumes that Angel considers a willingness to kill him a very good quality in his allies and friends.
In "Lonely Hearts", the way that the demon kills people is bad enough, but its final victim, the bartender, is shown to physically be falling to pieces owing to the damage that Angel, Kate and Doyle have inflicted to it, starting with flaps of skin coming loose and later entire chunks.
On that note, as the demon ejects itself from one host body to another, you can hear the sound of a painful last breath from the original host. Just how aware are its victims of what's happening to them? Probably best not to think about it.
From "In The Dark", Spike (probably best to clarify this is before his Heel-Face Turn on the original show) having Angel tortured is bad enough... but later on, it gets worse as the Psycho for Hire is revealed to have a fondness for killing children and eating them (and if they're lucky he hopefully does it in that order). Even worse, he gets a ring that makes him immune to daylight, which results in him going to the beach and walking right up to a group of girl scouts, creepy Slasher Smile all the while. Thankfully, Team Angel and Oz get there just in time to stop him.
Just to drive the point home, Cordy, Doyle, and Oz refuse to listen to Angel when he tells them to turn around to pursue the pedophilic vampire. Finally, Angel says, "He has a thing for children." Oz grimly looks back from the driver's seat and then pulls a U-Turn. It says worlds about the gravely terrifying nature of the vampire they're up against.
"I Fall To Pieces" takes the Stalker with a Crush concept and gives it a supernatural element, with a guy who can disassemble and reassemble himself at will. When he gets near his victim's house, he lets his hands go loose so he can open the girl's window and gets his detached hands to reach her crotch and start moving around. While it's not explicitly said what he did, Doyle immediately comments that he wishes he hadn't thought about that, and the girl's reaction goes from mildly aroused (probably dreaming of something else) to screaming in a terrified manner.
It goes From Bad to Worse when a cop runs into her house to help her, finds no evidence of what happened and is then promptly strangled to death by the two hands. Later, he proves there's nowhere he can't get into, as he manages to get part of himself in through the drain, then lets another part through the window and finally gets the rest in to chase his victim.
The reveal from "RM W/A Vu" as to why the ghost is haunting Cordelia's apartment is absolutely horrifying And I Must Scream material. The ghost is not haunting it because her son killed her, in fact quite the reverse. She tied him up behind a brick wall and built a new wall around him, ignoring his pleas for mercy and leaving him to suffocate. The start of the reveal itself is bad enough, as the pile of bones is enough to make all three of the gang freak out.
The Scourge from "Hero". The whole group of them are A Nazi by Any Other Name (even dressing in SS style uniforms and using Nazi Germany-era vehicles), with horrifying doses of The Social Darwinist. We see the result of one of their massacres when a terrified Doyle finds bodies of refugees piled up in a flashback. Even worse? Their device that they used to try and kill all the demon refugees works to indiscriminately fry all half-human hybrids. Doyle's death at its hands is not pretty.
In "Parting Gifts", Cordelia almost has her eyes cut out.
The way that Angelus' old student, Penn kills people in Somnambulist. Then, to make things worse, when the police plan to try and trap him, he takes the first move and casually enters the police department, causing Kate to freak out before he starts kicking their asses.
In "She", we see a guy being burned to death graphically, including his eyeballs bursting. To say nothing of the way that the females of the demon species are brainwashed.
"I've Got You Under My Skin" in its entirety, and notably any scene with the demon possessing the child. The shot of the kid trying to burn his sister alive is potentially even worse, and to make things absolutely horrifying this is done without demonic possession.
Wesley having a crucifix shoved into his neck, though he survives.
The demon describing how his time in the son's body terrified him, to the point he became a Death Seeker.
The demon's proclamation is enough: "Do you know what the most frightening thing in the world is? Nothing. That's what I found in the boy. No conscience, no fear, no humanity - just a black void. (...) I sat there and watched as he destroyed everything around him; not for a belief in evil. Not for any reason at all.
Angelus rising from the grave in the flashback in "The Prodigal", followed by him murdering everybody in his entire village.
The first time we see Angelus re-emerge in the present day is in "Eternity", and a good mix of Black Comedy and this trope, as the actress goes from pleading for him to turn her to screaming for mercy, all while he cracks bad jokes.
Cordelia tells Angelus that she keeps a cross in her desk, a stake in her bag, and has the office's drinking supply routinely blessed. This gives Angelus pause. Think about it for a second: Angelus is seriously considering that Cordelia (or anyone routinely interacting with Angel, for that matter) might be living in mortal dread of his return, because in all probability, they actually are.
The vampires in "War Zone" have a moment of Dangerously Genre Savvy when fighting Gunn and his gang. Instead of waiting for night-time, they dress themselves in Gas Mask, Longcoat style, and force everyone outside in broad daylight, kidnapping and turning Gunn's sister. Also counts as an Awesome Moment, albeit a horribly twisted one.
"Blind Date" has the moment when the blind asassin casually impales the kind old man, scaring all the children nearby.
There is an example of Mook Horror Show later on in the episode, when Angel turns the asassin's ability against her. We cut to her perspective as she loses sight of Angel, only to then get brutally beaten to death in the end.
Earlier on, Holland Manners changes from Affably Evil to Faux Affably Evil with horrifying speed, killing Lee Mercer for attempting to leave the firm for another company and take some of their clients with him.
Cordelia having a psychotic episode after the Vocah demon exposes her to the suffering of literally everyone in the world.
The Vocah demon itself is pretty bad, killing two members of The Powers That Be and Grim Reaper style dress code, though arguably the worst bit is seeing its real face, as it has a giant hole in the middle of it, filled to the brim with maggots.
Angel's stone cold stare just before he sets Darla and Drusilla on fire in Redefinition.
Angel furiously trying to smother Wesley at the end of "Forgiving".
He specifically makes sure that Wesley (and the audience) clearly knows and understands that it is Angel saying this, not Angelus, speaking in calm and friendly tones, right before pressing the pillow to Wes's face and ranting and raving at him, even foaming at the mouth a little.
The very end of "Awakening," when Angelus awakens. The episode ends with him turning his head and looking at Cordelia and the others, a Slasher Smile on his face and creepy laughter echoing throughout the jail cell. To say it's unsettling is a very large understatement.
Fred all by her lonesome being chased by Jasmine's Brainwashed and Crazy minions. Them all speaking in her voice doesn't help matters.
Jasmine's trueface, full stop.◊ Quite possibly the most horrifying thing in both series. Everyone's nightmares coming true? Angelus unleashed? The Wishverse? The Gentlemen? Being driven insane? The lengths Willow went to to resurrect Buffy? Xander's Eye Scream? The Scourge? Darla and Drusilla burned alive? Angel trying to kill Wesley? These are all low level scares compared to this.
The entirety of "Hell Bound" doesn't just have a nightmare scene, it's a freaking Nightmare Episode. The entire thing is spent on Spike slowly getting taken to hell, leading him to him getting surrounded by corpses and undergoing terrifying amounts of both physical and mental torture. It stands out that this is the only episode of either Buffy or Angel to have been originally broadcast with a warning about the disturbing content in the episode.
When we see all the victims of the torture, we see a guy who's sliced his own fingers off and (presumably) mutilated his face, a woman with no arms (this one actually is enough to get Spike recoiling in horror, dropping any of his Deadpan Snarker traits temporarily). Then we hear Parvayne speak for the first time, and it's a terrifying combination of Evil Sounds Deep and Evil Gloating.
Parvayne: She can't help you now, William. No-one can. *Evil Laugh*
Spike getting both of his hands cut off in "Damage."
Some may find the beginning of "Smile Time" to be especially so (probably due to the subtext of sexual molestation). This is the Breather Episode and even that isn't too tame.
Puppet: (talking to child through TV) You know Smile Time isn't free. Get over here and touch me!
(child gets up, touches TV)
Puppet: Oooooh, ohhhhh!
(child collapses with CREEPY Joker grin on face)
From the same episode, "talk to the hand!"
A Hole in the World, Fred's internal organs are liquefied in order to hollow her out for Illyria to use.
Drusilla's return in "The Trial". She walks into the room and doesn't say a word but you just knew badness was about to go down.