Not only is the man a known serial killer and cannibal, he's also a genius. That's not what makes him scary. What makes him scary is his Affably Evil personality. He has a charm and way of speaking to people as if he were engaging in everyday conversation with guests at a party rather than a trained FBI agent. Unlike the other inmates that Clarice encounters in the prison, Hannibal doesn't scream obscenities at her, he doesn't threaten to kill her, nothing like that. No, instead, he talks calmly to her as if she were one of his former patients or an old friend. The utter calmness yet meanace in his voice shows that he is not someone you want after you.
"What does your father do? Is he a coal miner? Does he stink of the lamp? You know how quickly the boys found you. All those tedious, sticky fumblings in the back seats of cars, while you could only dream of getting out. Getting anywhere, getting all the way to the F...B...I." Jodie Foster's shocked reaction is genuine and rightfully so. It shows that Dr. Lecter is not someone to be taken lightly and that he knows how to get inside someone's head.
Lecter's sense of smell. "You use Evyan skin cream...and sometimes you wear L'air du Tempsóbut not today." That moment is both weirdly erotic and incredibly creepy.
Lecter's horrendous taunts to Senator Martin in both the book and the film.
The (in)famous line "A census taker once tried to test me. I ate his liver with some fava beans and a nice chianti", and Hopkin's improvised hiss.
The scene where Hannibal beats the prison guard to death. Dr. Lecter's Nightmare Face during his escape is creepy as well. When hes bashing the guards' heads in he almost looks bored by the violence, he is truly emotionless and dead inside.◊ It's only outmatched by his brief Slasher Smile which is truly disturbing.◊ After an hour of pleasantries, winks, smiles and tears, it's unnerving to see this man, who we've heard horrible things about but haven't witnessed any of it, live up to his label of "true psychopath." Its debatable if he looks bored or if he is entranced by the killings, but its scary either way.
A later scene where other guards storm the prison room and find said prison guard suspended in a backlit crucifixion position with a big hole in his stomach, innards missing Together with the music, a Revolting Moment of Awesome... The whole part was terrifying. While transporting the surviving guard (who suffered severe facial lacerations) in the elevator, blood starts dripping down and he realizes he's on the roof of the the elevator. After he ignores the warnings of the SWAT team on the floor above, they shoot him in the leg without him even flinching. As soon as they open the hatch from inside the elevator, it cuts to the ambulance, where the guard sits up and we find out Hannibal had taken his clothes, and was using his face as a mask.
Lecter's Off Stage Villainy is terrifying in itself- from finding out that he attacked a nurse (seemingly for no reason) and that his heartbeat remained steady even "when he ate her tongue", to learning that his cell-neighbour Miggs killed himself by biting his tongue solely because Lecter spent the night whispering to him. a Hannibal Lecture that killed someone.
Buffalo Bill is also insanely creepy.
The part with Catherine in the well, where the camera pans up and you see bloody trails and the severed fingernail from the last girl who tried to climb out and failed. The book has the added note that Catherine remembered reading about a girl's broken fingernails before that in an article about Buffalo Bill and realizes who has her. In the movie, Ted Levine's completely inhuman and mocking screams just pile on the horror. What is particularly brilliant about this scene is that it shows a terrifying situation, something horrifying, sickening and disturbing... and yet something that is instantly quotable and amusing to almost anyone who has seen it. That's scary.