The wights and hollowgasts. There are several in-universe examples: the ones Jacob sees the night his grandpa is killed are literally nightmare fuel for him for months. Later when he is going through some creepy old photos he wonders what the point of some of them could be other than "fuel for nightmares".
Claire has a mouth on the back of her head.
Mother Dust's peculiarity is the ability to heal people and/or make them go to sleep using dust made from crushing up bits of her own body. Worse yet, the body parts she uses apparently don't grow back, and she seems entirely okay with this.
How poor Abe dies: sensing a hollowgast is coming closer and closer, knowing it wants nothing more than to kill and eat him, unable to get to the reserve of weapons he's stockpiled for this very moment. Even worse, it's his son and grandson who keep him from the weapons— unintentionally, of course, acting only because they think it's in Abe's best interests, but still. Knowing the monster is coming closer and closer, and not being able to do a damn thing about it— yeesh.
Primarily in the second book: being hunted down, knowing you'll be tortured and killed, by wights— who, at this point, are *everywhere*, including both sides of the military. Literally nowhere is safe.
Hugh's method of attack. Killing six men via covering them with bees and stinging and choking them to death, until their puffy, swollen corpses are the only things left.
Children who stay outside of their time loop for too long will start to rapidly age. Miss Peregrine mentions a little girl who wandered off while outside of the time loop, and aged thirty-five years in just two days.
The fact that (mentioned in the first book) the children actually have a game called 'Raid the Village' where they go around wildly destroying things with their powers and generally terrorizing people- because it will be completely forgotten and undone the next day. What makes it nightmare fuel is Enoch's implication that this is something the kids do to keep from going completely crazy from being stuck in the same day for (conceivably) forever.
Every day in the evening, bombs will fall, and the children treat it as a game.
In "A Map of Days", the children are taken prisoner by an insane peculiar girl with People Puppets powers who commands and abuses them like toys. This is bad enough, and it isn't helped by her henchman, whose first inclination towards all newcomers is to eat them. And the girl admits that she has done so in the past, but complains that doing so made her feel unwell.
In the books, Enoch's homunculi were just clay figures brought to life. In the films, they're ungodly Mix-and-Match Critters with baby doll heads, and he pretty much only brings them to life to watch them try and kill each other.
Freakier still, he implies that he's done this with actual human corpses.
From his parents' funeral parlor.
Olive's absolute glee at watching the homunculi brutally murder each other is also unsettling.
The design of the Hollowgasts. Slender Man with mouth tentacles is putting it nicely.
The entire sequence where a Hollowgast chases the children through the house. Miss Peregrine has been kidnapped, Miss Avocet is dead and Jake's father has no idea where he is or what's happening. These children are being hunted by a vicious monster and there's no one coming to help them.
Mr Barron's death. He disguises himself as Jake and gets his eyes slowly eaten by a Hollowgast, which can't tell the difference between Barron and the real Jake.
The day the loop was created, Victor was killed by a Hollowgast. His body lies in a room upstairs, never rotting due to the loop while the Hollowgast who killed him returns every day and if it isn't killed it will kill again.
