This is based on opinion. Please don't list it on a work's trope example list.
Nightmare Fuel / The Silmarillion
Everything about Morgoth/Melkor. The Satan of Arda, Morgoth engaged in Mind Rape without even having to think about it, and the only sort of pleasure he could experience came from horribly torturing and corrupting Elves and Men. In-universe, lesser Ainur were driven to insanity by his eyes.
Also the fact that Melkor captured many of the Elves and corrupted them, turning them into Orcs. What did he do to them?
Hurin: the mere thought of drawing Morgoth's undivided attention is alarming, but he keeps going at Hurin for years.
Just when you think that the good guys had won the battle, Morgoth unleashed the fires of Thrangorodrim unto them and to the nearby lands. Oh man! This is no joke, in fact this battle called Dagor Bragollach (Battle of the Sudden Flame) itself is such a nightmare into its own right. People burned left and right, and they were not only burned, but consumed by the lava itself.
How did Morgoth manage to provoke the Western host of elves & men into a rash attack before the eastern host could arrive? By having his orcs parade a prisoner of previous wars in front of the elves & men. Then they cut off his limbs & beheaded him right in front of his brother. And they also mentioned in passing that the same treatment awaited the other prisoners in Angband.
At the tail end of Unnumbered Tears, the Men of Dor-Lomin fought to the bitter end. In a final insult to their foes, Morgoth's orcs made sure to behead the fallen men & piled their heads in a great heap turned golden by the rays of the setting sun.
Ungoliant, as one would expect from a giant spider who brought Shelob into the world. A simple Maia, she drank the sap of the Two Trees and turned into the pure embodiment of darkness and gluttony, and became the Broodmother of all spiders in existence (sorry arachnophobes!). She's never satisfied, and isn't picky about what she eats. She would devour the entire world if given the chance. She almost ate Morgoth himself. And what's worse, the published version of The Silmarillion does not confirm her fate. So...she could still be out there somewhere...always hungry...
She utterly triumphed over Morgoth, making him scream with such pain that the mountains echo his scream forever. It took the combined efforts of every Balrog in existence to even drive her away from him, and none of this managed to actually hurt her.
Some earlier versions of the Silmarillion are even more chilling. Instead of being a Maia, Ungoliant was a primeval spirit of darkness from the void (essentially outer space). Perhaps there are others like her dwelling in the darkness.
What the imperialistic Numenoreans did while under Sauron's instruction to the peoples of Middle-Earth like the Haradrim, the Variags, and the Easterlings. We never get the details, but whatever it was, it so fouled their hearts with hate and pain that centuries later they worshiped and served Sauron just so they could kill Numenor's descendants.
Most likely they were being enslaved and sacrificed to Morgoth, as it is mentioned that Numenorians did that to men of Middle-Earth. Still a terrifying thought.
Ancalagon The Black. So humongously large, so large that he himself broke 3 mountains that are bigger than Mount Everest by his death. He has enough power to melt many of the rings of power. He is basically a flying natural disaster.
Imprisonment on the Isle of Werewolves. Trapped in a dungeon, with werewolves gradually picking the prisoners off. The Lay of Leithian even describes chains that eat their flesh.
Maedhros' torment on Thangorodrim. Morgoth catches him, tortures him, and then hangs him by the wrist from a sheer cliff face. It's not clear how long he was up there, but judging by other events in the timeline his imprisonment lasted for something like twenty years. It's so terrible that he can only beg for death when his best friend Fingon finds him. Then, of course, Fingon has to chop off his hand in order to get him down.