This is based on opinion. Please don't list it on a work's trope example list.
Nightmare Fuel / Sinister
Hanging Out '11
Every one of those damn reels. Which are every single doing of Bughuul.
Let's see, families including young children being lifted slowly off the ground by their necks while they twitch and spasm until they're all still. This scene (and the projector that played it) came from the nightmares of writer C. Robert Cargill after he first watched the American version of The Ring.
And last but not least, the damned lawn mower. Not even the movie thought it was a good idea to show that particular bloodbath.
Additionally, there's also that last-second Jump Scare, just because.
And we can only imagine what's on the newest tape... oh wait, Ashley was kind enough to leave some illustrations.
Bughuul's face's appearance on the surface of the waters of the pool.
On the laptop's screen, his image moved.
The way he does it, just casually looking at Ellison before turning back.
Bughuul appearing in the backyard. You saw it in the trailers, you knew it was coming, but it's nonetheless one of the most terrifying parts in the movie.
The ghosts themselves are deemed as this.
The scene where each one of the ghosts follow him around the house, sometimes close enough to touch, yet he can't see them. They cause creaks and other noises, and it seems like he catches small glimpses of them when they turn a corner, but the simple fact that in his eyes, there's nothing there, puts a whole new level of scare into Nothing Is Scarier.
The drawings on the wall, particularly the one of Stephanie.
The music that plays when Ellison burns the reels and projector and during the credits, courtesy of Boards of Canada. In the reel burning scene, you can hear high-pitched screams if you listen very closely. Also to add the films later on used ambient songs from Norway project Ulver, using certain songs like this.
The implication that the horror may easily continue, because if Deputy So-and-So starts investigating the killings following Ellison's death, he'll start searching for the case and once again plastering Bughuul's face all over the wall.
Or the professor Ellison was talking with online will, when the gruesome murder of a famous author's family hits the news and reporters start tracking down whether the research he'd been doing on his next book was connected to it.
Deputy So-and-So already knows what happened and has proven to be quite Genre Savvy. The rest of the authorities, on the other hand...
The overall sense of unease in the film, because there is a lot of movie shots were Ellison is alone in the night, with half the screen showing a darkened corridor. The audience keeps expecting something to suddenly crawl out of the shadows, and turning on a light or waking up his wife wouldn't have helped Ellison at all.
The sequel's reels don't seem to be any better:
Tied up in a flooded basement with live wires at the ready.
Hooded and hung upside down over a swamp filled with gators.
Being crucified and set on fire.
Tied to a dentist's chair and their teeth get mutilated with an industrial drill.
Nailed on a floor with rats being trapped under a hot container of some kind. Said rats escape by eating their way out.
Buried in snow up to their faces to freeze to death. The worst is the victim who hasn't died yet, and turns her eyes toward the camera in the final shot.
In a sudden swerve from the paranormal focus, Clint is one of the scariest elements, to the point that he counts as the film's Big Bad Wannabe. He's simply an abusive asshole of a husband who has every police offer in the state in his pocket, including the sheriff himself.
The scene with him at the dinner table, where he suddenly snaps and assaults his son when he won't eat, shoving food over his face and angrily berating him.
We finally find out what happens if a child fails to complete a tape, as Zach does in the finale after Deputy So-And-So breaks his projector. Bughuul walks up behind him while he yells at the ghost children, rests a decayed hand on his shoulder, and slowly rots him away until he's nothing but a skeleton. Then he torches the house and moves on.
Zach in general is unnerving from the second half of the film. While early on he was seen as an average, if angry kid, he really goes off the rocker and begins to show sociopathic levels of cruelty and assholishness, even going so far as to state that his movie would "be the best one yet", talk about Enfante Terrible.