Please don't list this on a work's page as a trope. Examples can go on the work's YMMV tab.
Nightmare Fuel: Myst
If you read the books in the library, you learn that the worlds weren't always empty. Sirrus and Achenar murdered every last inhabitant of 3 populated worlds. These are the only worlds that they haven't completely destroyed. All of those burned out books on the shelf used to describe the other ones.
If you get one of the bad endings, the game doesn't stop. You just remain trapped forever, until you turn the game off.
Myst is fairly bright and happy, with nice ambient music. But the sense of loneliness can get to one after a while. Then there's Achenar's rooms in the Stoneship and Channelwood Ages, which are filled with torture devices. The music in Achenar's Cache has probably the scariest music you'll ever hear.
Try the bad endings of Myst with the brothers, where you return the final page to Sirrus or Achenar and get trapped forever in that brother's book. The worst part of that is that the brother laughs at your foolishness and then proceeds to rip the pages out of your prison, one by one by frigging one, while tormenting you the entire time.
The hologram of Achenar trying to speak the Tree-Dweller language in the Channelwood temple. As if the music in there, the altar that eats its sacrifices, and the masks on the wall weren't creepy enough. The music, by the way, is a remix of Achenar's leitmotif.
Atrus himself is kind of horrifying in Myst, when you realize that He intentionally trapped his two sons forever in a hellish blank landscape of nothingness forever, and when you free him and return to the library all that's left of his sons are scorch marks from where he burned them alive.
The edge on the Atrus trapped his sons thing is somewhat dulled when Atrus explains that he never intended his sons to become trapped in them in the first place, and in order to even use the books Sirrus and Achenar actually had to trap Atrus and Catherine first:
It's even more dulled by the retcon that their prisons are in fact full Ages with no Myst book (which you visit in Myst IV: Revelation), and he merely destroyed the linking books in the library
But nonetheless, where they were trapped all alone for over twenty years. The already unstable Achenar had a full-blown mental breakdown and slaughtered half of the creatures on the island before regaining his senses, several years later.
Riven accomplishes scary in a game where you know you won't be attacked. Ever. Using only music and lightning. Scary!
Except that you get attacked in Tay and even killed by Gehn if you mess up badly enough.
There's also the bad ending of Riven where you trap yourself in the trap book while you're in Tay. Two rebels find your book, and the lighting implies that they're burning the book with you inside of it. They are quite literally burning you alive.
Even when you do the right thing with the trap book, the implications are disturbing. You use the trap book in Gehn's presence, hoping it will trick him into thinking it leads back into D'ni. Through the window, you see him open the book, dressed fully for the D'ni climate, and check his rifle to make sure it's loaded— and then he lays his hand on the window, fully expecting to find (and kill) you and Atrus on the other side.
A strategy guide for the game compounds on this, being written as a first-person account of the events of the game. As Gehn uses the trap book himself, the protagonist describes hearing Gehn's despairing scream as he realizes his mistake, and knows full well where he's going and what he's in for.
In Riven, there are the wahrks (no, that's not a misspelling). The name is a portmanteu because the creatures are as big as whales, and as vicious and deadly as insane sharks. As you progress through the game, you see hints of the creatures' all over, and you learn that they are used in Gehn's gallows, where they rip apart and eat people that have made Gehn mad. And where do you learn this little tidbit? In a school building. On a toy. Used to teach kids their numbers. And it doesn't end there. Eventually, you reach an underwater lab on one of the islands, where you have to solve a color puzzle using totem poles which sit in front of a massive underwater porthole. And guess what happens when you click the red one? The view tilts up and reveals a warhk that swims in front of the porthole and bares its teeth at you. Click it again, and the wahrk returns, but is clearly in a bad mood — he was expecting to find a meal dangling from the gallows above. And if you press the button a third time, nothing will happen for a few seconds... before the now extremely pissed-off wahrk suddenly charges at you and rams the window face-first. After all the the lonely atmosphere, and all of the depictions of the wahrk, a real one is so completely frightening that at least one player shut off his game in abject terror.
Hard to get across now, but Riven was particularly shocking for many Myst fans because they were used to the deserted, almost immobile views of Myst, and then suddenly you have a situation where characters can walk into view and start talking to you.
Including a scene in which you enter one screen in the treetop village, where a little girl is watching you then runs away, if you aren't expecting it, it can be a decent jump scare.
Gehn's lab is quite eerie, with the foreboding feeling that Gehn might walk in any second.
Myst III: Exile
Just about everything Saavedro does is creepy to the max.
Saavedro's entire life. Imagine the terror you'd feel after being stranded on J'nanin when Sirrus and Achenar burnt your linking book home. But there's a ray of hope, when you realise the function of the Age you're on! You figure out Atrus' lessons, listen to his messages spoken with fatherly warmth towards the two monsters who came to destroy your civilisation, making your way through each age in turn. And after all that, when you finally get back to Narayan... you find the way home blocked by a force-field you cannot circumvent. Now imagine spending twenty years confined by yourself, to all those ages, with your home just out of reach? No wonder he went insane...
That painting in Amateria. Not helped that the individuals portrayed in said painting are the already frightening Sirrus and Achenar.
If you watch Saavedro pace in the tower before you enter, you can see him messing with the telescope puzzle things. Now, if you can easily see half of J'nanin through those things, especially the areas you've already been in... what makes you think that Saavedro couldn't see you walking around down there?
He likely didn't use them to spy on you. Otherwise, he would have known you weren't Atrus.
One of the endings: Imagine a person trapped on a balcony maybe 6 paces long in between two completely impenetrable walls, behind one wall is the system used to control the wall, behind the other is this person's home civilization and family, both of which he had been convinced for years were destroyed by two people he had trusted. There is absolutely no way for him to pass either wall despite him being literally feet away from a gondola that would take him home. Worse of all, he had to watch a person, who could have released him by switching two levers, teleport himself away to another world despite him begging and pleading on his knees, and knowing that there is no possible way for anyone else to ever find him because the book that makes up part of the only path to the world where he is has been burned. Now consider that you are not the trapped person, but the person who left him there and that you know he is a man who believed everyone he ever knew and loved had been murdered by the two selfish sons of his friend, and that the only places he could go for literally years were four islands that made up a puzzle for young children with nobody in them but himself.
Myst IV: Revelation
Myst IV features you trying to rescue Atrus' kidnapped daughter; you can view her fairly disturbing abduction and attempts to escape in flashbacks. What the badguys have in mind for her is scary in and of itself, not to mention the Creepy Child moment near the end.
Also in Revelation, there is the scene where the camoudile goes right over The Stranger after the latter hits the ground (after being kicked in the air by a zeftyr).
Related to this, there's a weird "Let's choose a spirit guide!" ceremony, just before the part when you go and look for the aforementioned guides. There are three Serenian acolytes, each meditating at a different shrine and in different outfits. And they all have creepy, placid masks on. After the strange ceremony (which involves sticking your hand into a silver liquid that burns your handprint onto a sheet of cloth, along with strange glowing lines that correspond to the spirit that chose you), A fire acolyte comes up and gives you the stock pronouncement of which spirit chose you. Then the camera moves and HOLY CRAP THERE'S A FREAKY MASK PERSON RIGHT THERE!!!!! Of course, that didn't help the frustrating and fiddly puzzle right after that involving getting an offering to said spirit so that it'll transport you to the next puzzle. Serenia was nasty like that. We did get some cool music on the first link, though.
Myst V: End of Ages
Escher's hut in Noloben, adapted from a building the Bahro made. There's a cage in it he uses to keep Bahro for gruesome experiments to find out things like their brain structure (he may well prevent them from linking away by utilising their fear of snakes). Knowing his hatred for them, they may have been alive while he did this. The cloth on his shoulder is similarly bad; he's able to link using it because it's a piece of Bahro skin.
There's a wahrk skeleton on the other side of the mountain from Zandi's trailer. You'll want to wet your pants retroactively when you realize just how very damned big it is. (The mandible is standing upright. It's big enough to walk UNDER.)
Perhaps the creepiest thing ever was in Uru, in the Teledahn Age. The slave areas below the hut can scare the shit out of you, especially if you've read Book of D'ni and are wondering if this is actually a Terahnee age.
In URU Live, near the end of the game's second run, there was a storyline where Willow "Wheely" Engberg was trapped in a room with a Bahro, a race of creatures enslaved by the D'ni centuries ago. As the players watched the live roleplay, Wheely became more and more frantic, screaming over the comm system as the Bahro repeated "Noloben" over and over again and advanced on her, and when the rescue crew reached the room, well...
It gets even worse when you recall what the whole plot of Uru Live basically was - a war between two factions of the now freed Bahro, which the DRC, Yeesha, AND the explorers of the cavern are all caught up in. Of those two Bahro factions, one uses their newfound freedom to peacefully try to engage with the explorers, while the second faction vows vengeance for their previous suffering and isn't shy about killing people, hence Wheely's fate. The scary thing? You literally cannot tell the difference between which Bahro want to harm you and which are on your side. Even scarier is that the 'bad' Bahro's leader looks like some sort of zombie version of a Bahro, with a huge scar where a patch of his skin was ripped off during Esher's experiments on him and bright red eyes.
The cloud of death that all but wiped out the D'ni in Myst: The Book of Ti'ana.