This is based on opinion. Please don't list it on a work's trope example list.
Nightmare Fuel / Lost Tapes
The freakin' Reptilians! They can take on any form, are smarter than us, and have infiltrated pretty much everywhere. If you get marked, they can find you no matter where you go. Then they wrap you in this weird plastic stuff that asphyxiates you but also keeps you alive and unconscious. They clearly need live humans for something, but for what?
The Monster of Monterey episode. If being stuck out at sea isn't enough for you, how about seeing it happen to your friend on a live recording? Throw in you being knocked off your boat, with no way back on, and a plesiosaur dragging you under, and you can't see it, and you've got prime Nightmarefuel.
A blink-and-you-miss-it example; as they show a POV shot of Sharon slowly drifting away from the boat, she suddenly gasps and shouts "what was that!?" before being pulled under...
Some episodes manage to be truly scary: Zombie has some real jump-inducing moments. Poltergiest, Strigoi, Alien, and especially Devil Dragon tend to stand out.
Poltergeist especially, particularly the feelings of hopelessness that come up as the documentary crew is killed one by one, especially the circumstances under which they all die: the woman in the wheelchair is thrown against the wall and convulses as the poltergeist possesses her to tell the others to leave; the cameraman is killed, blood comes from his eyes, and he just stays standing there like he's in a trance rather than dead; and their asthmatic leader is lured upstairs by what sound like cries for help, only to be confront by the poltergeist—which only he can see—and is stabbed to death. Holy fucking shit.
The clincher? By the time the cameraman dies, the leader gives up. He immediately starts screaming "Okay! You win!" to the poltergeist in a terrified voice before desperately trying to escape. But the poltergeist isn't satisfied with them leaving the normal way. No, he wants them to leave in body bags. And that's exactly what he gets.
Really? Because I remember the poltergeist causing a bloody message to appear on a mirror to the tune of "too late". The ghost gave them a chance to leave, they refused, and it became a matter of "you had your chance, now you die."
Just to add to the terror of the scene, as Jeremy (the leader of the documentarians) tries to leave through the front door, but it's locked. No problem, right? He's inside, so he has control over whether the door is locked or not, right? Nope. The poltergeist keeps locking the door, and Jeremy is clearly becoming more panicked as he undoes the lock only for it to close again seemingly all by itself the second he takes his hand away. Much like in other paranormal horror stories, it's a very mundane thing which becomes much, much scarier by the fact that you can't see what's doing it.
Owlman had some nightmarish moments, including how the girl seemed to draw him, and see him everywhere. He wants girls for eating, and who knows what else. Hellhound was worse.
The part where he ripped the old lady from the earth, the one he had been pursuing, was also rather frightening. Leading to many Owlman related nightmares for me.
That and the impending Fridge Horror that the little girl could be next.
Sue Ann: It knows you're here to stop it. And it's coming back to stop you.
Let's not forget Wendigo, as seen in the above Trope Image. Not only was it surprisingly good special effects and acting, the clips of him approaching the Despair Event Horizon after a week of no food hit really uncomfortable places. Paraphrased conversation:
Girl: We can't just leave her [their injured friend] here!
Guy: I'm not saying that, but we can barely look out for ourselves!
Girl: Well, what other plan do you have?!
Guy: (guilty silence) How long has it been since you had something to eat?
Girl: ...You're sick.
That ENTIRE episode counts as High Octane Nightmare Fuel.
Good lord, the fucking Wendigo. YMMV be damned, that thing is one of the scariest monsters on the show. To top it off, it has what looks like A DEER'S SKULL as a head. Sweet dreams.
The thing that makes it truly scary is that Wendigo psychosis is quite real.
Knowing the myth of the Wendigo makes it even worse, and they tell you it during the episode. Pretty much, it's an unstoppable killing machine that can never be full. It'll stalk you until it gets you, no matter how much it's eaten it'll just keep coming after you until it's got it's claws on you.
The scariest parts of the episode were the climax and ending: Imagine being trapped in a dark cave at night with the Wendigo coming towards you and the ending has the Wendigo staring right into the camera.
To paint a picture; imagine a cross between a vampire bat and an ape with extremely sharp teeth and claws and a Hair-Trigger Temper. Now picture it chasing you, with the intent to render you hollow of blood. Oh and sunlight? Yeah it doesn't do squat to it.
While possibly driven less by a rational choice and more by panic, one of the older kids (incidentally the same one that accidentally broke the kid's leg) suggests to the other that they leave the young boy to die out there.
While it's far from accurate, the implication that the Thunderbird is an Azhdarchid pterosaur that acts like a shrike. Imagine it...a flying, giraffe sized monster that can eat you whole, but instead prefers to kill you, eat part of you and then impale your dead body on a branch to eat you later.
Dover Demon, for all its Blair Witch-iness, taps into the fear of something with gangly limbs and giant eyes coming at you in the dark. And then the night vision comes on.
This is only nightmare fuel if you think about it, but some descriptions of what happens to some victims can be even worse. For example, in Lizard Man, the news crew is stated to have been partially eaten after the Lizard Man got them. And also in Devil Monkey, at the very end, it's stated that the bodies of all the characters presented in the episode were found—except for the teenager who reappeared halfway through because he survived—but they were so badly mutilated that they could only be identified by DNA and dental records. Just think about that for a moment. *shiver*
In the Chupacabra episode, there's a moment where the little girl drops her camera which then catches her running away. She looks back, screams, and runs even faster. When it cuts to the pre-commercial title card, we hear her parents screaming and crying and the chupacabra snarling. The implication? That the little girl saw her parents getting mauled to death by that thing. Definitely NOT something a little kid should be witnessing.
Not only that. The ENTIRE episode can count as Nightmare Fuel. For one, we never actually see the Chupacabra. We just hear it´s horrifying sounds and the aftermaths of it´s rampage, but that only makes things WORSE. Also, even though the beast is shot at the end of the episode, there is nothing that confirms it´s death, so for all we know it could very well have gotten away and still be at large...
One can say the scariest are the ones that is most likely to happen in real life. Like the "Wendigo" episode.
If you have any fear of insects, even in the least, "Death Crawler" will make you hide under the bedsheets.
Two scenes in particular: you see lightning flash over the tent and the shadow of a giant centipede is shown. The kicker, IT REALLY DOESN'T LOOK LIKE FAKE CGI. The next scene is marked for spoilers. The couple is on the boat, happy they got away. When suddenly the man looks to the side with an expression of horror...cut to days later when it's stated that their bodies were found with strange marks.