This is based on opinion. Please don't list it on a work's trope example list.
Nightmare Fuel / Alastair Reynolds
Captain John Brannigan's description.
His head, by contrast, was almost normal. But only by contrast. Red multifaceted cameras were crammed into the orbits of his eyes. Tubes emerged from his nostrils, curving back around the side of his face to connect to some unseen mechanism. An oval grille covered his mouth, stitched into the flesh of his face. His scalp was bald save for a dozen or so matted locks emerging from the crown. They were tied back, knotted into a single braid that hung down the back of his neck. He had no ears. In fact, Scorpio realised, he had no visible orifices at all. Perhaps he had been redesigned to tolerate hard vacuum without the protection of a space helmet.
The Melding Plague. Affected people who had head implants suffered head explosions. Buildings would reshape themselves, trapping people in the walls, which morphed to fit their screaming profiles. It is unknown if said trapped people can be revived.
What happened to the Eighty (except for Aurora)—their minds were uploaded, completely frying their brains in the process, and then they all just started failing. Some were frozen in time to stop the degradation process, but it is similarly (as of The Prefect at least) unknown if they can be brought back.
Skade's torture of Clavain. Made worse by the fact that first, he voluntarily stops his neural implants from blocking the pain; second, that it is not described—the story picks up afterwards; and third, that Skade made Scorpio, who had become close to Clavain, carry out the torture just so she could enjoy it more.
The torture game from which Scorpio escaped.
In The Prefect, some simulations of people killed in an attack on a space habitat are infected with a virus that deletes them in the course of the subsequent investigation. They can feel themselves being deleted.
Also in The Prefect, the VTs—"Voluntary Tyrannies". Because the conditions on such habitats arose out of democratically-decided rules by that habitat's inhabitants, the Panoply is powerless to effect any changes.
Hell Five deserves special mention, being that it's completely normal except for the fact that one citizen selected at random to be tortured to death in front of cameras. When he dies, repeat.
Arguably, the Greenfly, what with their "use-any-available-material-to-make-green-stars" mentality. An additional level of dread added by fact that even Nestbuilders weapon not able to defeat them. And an implication that they will destroy our universe in the far future.
To elaborate, the Vorgs are carnivorous cyborgs that eat brains to compensate for their failing circuits in the Zones. They are wrapped in perpetually rotting muscles exposed to open air, and have to kill (very brutally) to replace them.
Sky Haussman's clown.
And his dolphin...
Inertial suppressor malfunctions have a nasty habit of erasing people from recent history, rewriting the time-line so that they died years prior. Only close bystanders have any recollection of the person who vanished, and are assumed to be delusional.
The scrimshaw suit in Absolution Gap is quite the terrifying torture device, especially to anyone with the slightest claustrophobia. The suit is a human shaped coffin with complete life support, only allowing the tiniest of movements inside it but allowing its occupant to live for years with little or no contact with the outside. Having one character's lover imprisoned within the suit and hearing her slowly freak out at her predicament doesn't help, either.
Cultists in Chasm City use indoctrinal viruses to spread their faith—brain-altering diseases tuned to produce appropriately mystical hallucinations, and even physical manifestations such as stigmata.
The entire universe. The sheer level of hatred it holds for all sentient life made The Other Wiki compare it to Lovecraft. And not the Lite kind, by any means.
Also from The Prefect, the erasure of the beta-level simulations. Imagine knowing that your mind, your memories, the very essence of your being was being erased forever...being fully cognizant of it and being unable to do anything to stop it.
Diamond Dogs features a mysterious "Only Smart People May Pass" tower; the maths puzzles inside get progressively harder, and the penalties for wrong answers get progressively more horrible. In the end, it turns out that the strange rocks the protaganist found on the plain around the tower were actually the shredded remains of his friend, who was diced by the tower and ejected like confetti. Repeatedly. The top of the tower is not worth it.
Multiple short stories from Reynolds' Galactic North anthology are frightening. In the Nightingale story, the protagonists set out on an exploration mission of a supposedly abandoned medical spacecraft that ends up being a Living Ship. In the end the team is surgically fused by the ship into a single Body Horror being to send message about the horrors of war.
Grafenwalder's Bestiary deserves a special mention too.
Tanner Mirabel's dream about being eaten alive by the hamadryad.