Yu Yevon. He created Sin as sort of an 'object lesson'; it was never meant to be permanent. He also created Dream Zanarkand. Unfortunately, he did both at once and it wiped his mind clean, overwriting it with his instructions for Sin. He can't stop. Ever. He's been rendered braindead to the extent that he's essentially a machine following a program. He died at some point. He never noticed.
Fiends used to be people. For added Squick, consider that when you hear Wakka and Rikku's humanitarian jokes about eating Behemoths and Dark Flans.
Sin itself. A human soul, changed into a monstrous summon to defeat the previous Sin, then forced to kill the person that summoned them, someone they loved enough to die for. And this goes on for a thousand years.
The summoners go on long, dangerous journeys with their guardians and cheerfully - almost to the point of brainwashing - accept that they will die by the end of it and will need to kill one of their beloved guardians. What they're not aware of is that all of this will only serve to continue a vicious cycle. And people celebrate it; even children seem excited about it. Made worse by the fact that half of the highest ranking people in the church are dead and unsent, give no care to the sacrifices and pain of the living, AND are only two pegs above being a Nietzsche Wannabe convention.
The knowledge that one of your guardians has to die is withheld until you get to Zanarkand, unless you are a Maester or another prominent Yevonite.
The Dark Aeons, corrupt versions of Yuna's regular Aeons. Specifically:
Dark Shiva. She will give anyone nightmares with her ridiculous speed and attacks.
Dark Yojimbo. You're just exploring the Cave of the Stolen Fayth, when suddenly it starts CHASING YOU.
Anima◊. Just...every last thing about her. A massive, twisted humanoid giant, chained and hideous. Its appearance is disquieting, but the real nightmare fuel comes when you learn what Anima is: Seymour's mother, who gave herself as Seymour's Aeon because she saw no other way to redeem Seymour in the world's eyes - and later, when Seymour made his Face-Heel Turn, she couldn't stop him from using her as a weapon. Now look at Anima again, and see the pain and helplessness in her face...and then watch her Overdrive, and see her anger...
Seymour summoning her in Luca. When she kills the monsters in the arena, she's crying tears of blood! Her mannerisms are also pretty heart-rending in a disturbing sort of way.
Watching Crusaders get reduced to black particles at Operation Mi'ihen is extremely disturbing. Then Tidus does their autopsies if you examine the bodies.
Made worse by how quickly it happens. One moment you hear the telltale sounds of battle, and then those sounds stop without so much as an echo as crusader and Sinspawn alike are vaporized. It's practically one of Sin's defining moments - a testament to how insignificant any human (besides a summoner) really is.
The Chocobo Eater. The idea that there is something out there that eats chocobos is terrifying, and just before the fight, it picks one up.
Once you finally get inside Sin, there's a cutscene where the characters marvel at how surprisingly beautiful it looks in there...then suddenly it turns dark, an Evil Laugh is heard and you see a brief-but-horrible extreme close-up of Seymour, complete with Giant Eye of Doom.
Final Fantasy X-2'sBonus Dungeon has Elder Drakes which can wipe you out without breaking a sweat unless you've been Level Grinding like mad. Normally, this wouldn't be a problem, as there's always the Flee command... except, it doesn't work on fiends you can see before you fight them. You have to time your run perfectly to avoid them, and they keep jumping out at you - and the noises they make as they pounce... *shudder*
Listen carefully to the Hymn of the Fayth in each shrine. Near the end of the game, it becomes clear that the singer is the Fayth itself. Now listen to the version that plays in Yunalesca's chamber. It's not only being sung by a full chorus, but they're off-key. It's all the guardians who sacrificed themselves to create the Final Aeons. And they're in despair.
Even worse is the version that plays before you fight your Aeons + Yu Yevon. It's the same as the previous, except even more distorted, with a creepy echo effect.
Occasionally in the Sanubia Desert, you run across a GIGANTIC SANDWORM that is only rivaled by the Nintendo 64 Peahats for scariness.
Talk to Maechen at Mount Gagazet after getting the airship. If you take the time to listen to his exposition and put things together, you realize that Yu Yevon basically rules the entire world with the power of fear and death...and he's been doing this for a thousand years, with no one able to stop him completely until Yuna's party comes along.
Seeing the characters get Swallowed Whole by the Worm type enemies is really disturbing. Seeing them get regurgitated back out is even more disgusting and nightmare fuel inducing.
Freaking Greater Malboros. Giant desert worms are relatively common. Giant flesh-eating tentacular blobs with dozens of eyes? Not so much, outside H. P. Lovecraft.
Malboros are a common enemy across all FF games. The regular Malboros are at least manageable, as they attack normally and only use Bad Breath about 1/3 of the time. The Greater Malboros found in the Omega ruins always strike first (unless you have someone specifically equipped with a First Strike weapon), and always open with Bad Breath. And their Bad Breath attack causes many more status ailments (including ones that make you lose control of your character, like Confuse and Berserk). Pretty much the only guaranteed way to survive one is to have at least one character have confuseproof AND berserkproof armor on, or to have a first strike weapon and the flee command.
Geosgaeno, who terrorized Tidus at the beginning of the game and nearly succeeded in swallowing him.