Spider Swarms and Leech Swarms in 3.5 are terrifying for low-end characters: They automatically hit any enemies they're touching for non-trivial damage, are resistant or immune to most normal weapon damage, stack poison on top of that and make it so that a character can't actually run away because he's covered with spiders that are biting him to death. Warlocks get the option call such swarms at will, right at the first level if they want.
In Dungeons & Dragons 4th edition, Needlefang Drake swarms are Demonic Spiders to low-level characters. They have the ability to knock down characters, and then deal extra damage to characters who are knocked down in addition to the standard swarm ability of attacking all adjacent enemies at the beginning of their turn. Additionally, they take half damage from most attacks, and it is likely only one character in any given party has a reusable attack which deals decent damage to them at the very low levels you fight them at. One is dangerous, but managable; two is likely to result in player death. Three or more are very likely to kill the entire party. While trivial later on due to being low-level enemies, to level 1-3 characters they're nightmarishly deadly.
Skirmishers in general, and flying ones in particular, tend to be this to PCs of all levels. Their speed means it's difficult to focus fire on them, and they often have the ability to ignore or slip out of any status effects the PCs throw at them to try and lock them down.
The Bebilith and Retriever are actual Demonic Spiders, though the level 18 Solo Bebilith doesn't really qualify for the trope. The level 27 Retriever fits the quite nicely for Epic-tier PCs, and even comes in an upgraded boss-fight package in the form of the 30 Solo Retriever Holocaust.
See the quotes page for Instant Death Radius for a monstrous crab that will murder any party that tries to take it on at its recommended level.
In older versions (i.e. pre-4th edition) powerful undead were a nightmare. They were often immune to non-magical weapons, you could not get close to them without risking them Level Draining you to death, and if you died to them, you could often count on joining them, making the battle that much harder for whoever was left. A powerful cleric and his power to turn and destroy undead was utterly essential to survival against groups of them.
Incorporeal undead in 3rd edition, as they are immune to non-magical attacks and have a 50% chance to avoid most magical attacks. Special mention goes to the Allip: its Challenge Rating is 3, which means a level 3 party can be expected to run into one— probably before most of the party has magic weapons. With its Turn Undead Resistance, a level 3 Cleric is unlikely to be able to hurt it. Its attacks deal ability drain, which can kill much faster than normal HP damage and which a level 3 party is unable to heal, and each hit increases the Allip's current HP. Finally, its mere presence has a good chance of mesmerizing half the party for about half a minute.
Rust Monsters. They don't do much damage, and are easy enough to kill, but will go directly for whomever has the most metal - usually your plate wearer - and try to dissolve it all with a touch attack, something that's usually quite effective against plate wearers. And hitting them with something metal will cause that to dissolve too. Even magical items can be effected. This is a game where losing your precious +4 Armor is a bigger problem than dying— if your party survives the encounter, you can be brought back to life for 5,000 gp, but most magical weapons range from 8,000+ - 200,000+ gold, with magic armor costing a bit more than half that. Furthermore, replacing those items will take time, which is something you don't have in a dungeon crawl. Add that up, and that means a Rust Monster has the potential to cripple a party so badly that they can't do anything to fight back.
The tentacled Cthulhumanoid Mindflayers possess inherent psionic prowess, a hard to resist instant-stun at-will Mind Blast with a huge radius, and their ability to eat your goddamned brain is combined with ludicrous spell resistance and saving throws. Furthermore, as D&D goes on, they get more and more variations and options because they're such an iconic monster. How about fighting mindflayers with sorcerer/psion levels that can dispel your freedom of movement, or mindflayer liches with damage reduction, or maybe the batshit insane spiked-chain wielding monsters that are the Mindflayers of Thoon?
Will-O-The-Wisps are death machines to lower level parties. As they have natural invisibility, fly, very high AC, are immune to magic, and primarily make touch attacks. Unless you are high enough level, make special preparations, or have a lot of Magic Missiles (one of the two spells which are excepted from its immunities).
Swarms can be this to low level players, since they're immune to basically any type of damage that isn't an AOE spell, but they're typically Goddamned Bats.
Almost any monster with the Grab special ability, which allows mobs to grapple you. Bonus points if they have the Constrict special ability, which means they will do extra damage just by having your grabbed. The worst are usually plant type mobs, since they have a lot of immunities, high CMB, and are virtually impossible to detect in their natural environment until they attack. Examples include Shambling Mounds.
Ornithopter. Useless on its own, but with an Unholy Strength or Rancor, we're talking about two damage every turn. Played on the first turn.
Squee, Goblin Nabob. Not to attack, mind. Squee works differently. Squee can be discarded or played and sacrificed. Then he returns to your hand. And he's a goblin, so all those "Sacrifice a goblin..." cards qualify.
Any Rebel. Yes, all of them. Their abilities allow them to fetch more Rebels, which allow them to fetch more Rebels, which allow them to fetch more Rebels...
Rootwater Thief: Flying creatures can only be blocked by other flying creatures, and if he connects, he can remove any card from a player's deck, crippling many deckbuilds.
Voice of All. It's a small creature with flying, but when it comes into play, it gains protection from one color. Not just a specific color, however, but any color. Therefore, Voice of All could simply continue smashing in the face of any monocolored deck with impunity, or worse, create a near impenetrable defense.
Disciple of the Vault is a dirt-cheap common creature that makes the opponent lose life whenever its controller loses an artifact. Since the life loss was unpreventable, there were a ton of cheap artifacts, the creature itself was hard to kill, and its power stacked with multiple Disciples, Disciple of the Vault was one of the most aggravating creatures in the infamous Affinity decks due to how difficult it was to stop.
Bloodbraid Elf is a cheap creature with 3 power, haste, and the "Cascade" ability, which gives whoever plays it another cheap spell for free. So it hits hard the turn it hits play, and will probably hit hard again on that same turn. The deck archetype it was most often used in— Jund— used the Elf to dig up a Sprouting Thrinax, Blightning, or Maelstrom Pulse for free, effectively netting you two creatures, a shot to your opponent's dome and two cards out of his/her hand, or a quick Kill 'em All switch AND three damage for the price of one. In any other deck, that's useful. In Jund, it's deadly. This got so bad that Bloodbraid Elf was banned in Modern to prevent Jund decks from dominating the format.
Scars of Mirrodin has Plague Stinger, a small second-turn creature with infect— an ability that allows you to win by piling 10 or more poison counters on an enemy. Normally, this could be solved by blocking the creature; however, Plague Stinger has flying, making it especially hard to stop. This means every turn, unless you get some removal or a blocker with flying or reach, the Stinger is going to end up biting you to death, piece by piece. Throw in a pump spell or some Proliferate, and Plague Stinger goes from moderately dangerous to a game-ending threat that has to be answered.
New Phyrexia one-ups the Plague Stinger with the Blighted Agent, a blue card with the same converted mana cost and Infect ability. It can't fly, though. But it doesn't need to: it can't be blocked by anything.
In a similar vein to both, there's also Inkmoth Nexus. You can play it on the first turn and by the second turn, it can attack (making it one turn faster than Stinger). It's stats are identical to Stinger when it's a creature. New Phyrexian managed to up this threat by introducing Mutagenic Growth, essentially letting you whack at the opponent for up to 9 poison counters on the second turn. Bear in mind that you only need 10 to kill someone. This thing is also uncounterable, as it is a land coming into play. This means you have to use a kill spell on it while it's a creature or one of the expensive (mana-wise) land destruction cards, making it even harder to get rid of.
In the 1978 Avalon Hill board game Magic Realm the giant bats are among the most dangerous of all monsters. They are too fast to run away from or hit reliably, kill unarmored characters instantly, and wound to death those with heavy armor.