In the movie, the rhino is quite scary. Even worse is the mechanical shark. The dead pirates can be pretty scary too, right at the very beginning of their scene.
The premise of the book is Nightmare Fuel. Sure, James's insect pals look sweet and cute in the illustrations, but would you like to come face to face with a human sized centipede◊ or spider in real life? They're actually less anthropomorphized than, let's say, Disney's Jiminy Cricket who is essentially a little green-skinned man with no ears.
The scene at the end where they show up in a desperate last attempt to bring James back. You can practically feel his fear!
James's nightmare in the film. The animation is just creepy.
The Old Man is extremely creepy in the book. His first spoken line is: "Come closer to me, little boy. Come right up close to me and I will show you something wonderful." Throughout his meeting with James he acts rather unhinged and slightly sinister, and it's impossible to guess his real motives for giving James the crocodile tongues — when James loses the bag so the magic works on the peach tree and the bugs instead of on him, there's the sneaking suspicion that he may have escaped a fate more horrible than anything Sponge and Spiker could have done to him.
In the film, the Old Man is heavily toned down; even if a bit of the creepiness remains, he's a lot gentler and is explicitly shown to be benevolent (especially when he reappears, in silhouette towards the end to tell the New Yorkers to "let the boy speak!"), and is revealed to have been the narrator of the story all along. This does not happen in the book.