Barbarians. They aren't much trouble early on, when they're animals (which won't enter your cultural borders) or Warriors (which are only dangerous to Settlers or Scouts). Then they discover bronzeworking. Barbarian Axemen will wreck your infrastructure like nobody's business, are a pain to take down in melee, and tend to come in packs - and they usually bring at least one Archer with them just to keep you guessing. The really irritating part is that, after about 800 BC on any difficulty harder than Settler on the expansion packs, these guys never stop coming. Build the Great Wall as soon as you have access to it, or, barring that, keep Archers near your important tiles.
Civ IV Axemen are easy to deal with in the Warlords & Beyond the Sword expansion packs, since chariots get a giant attack bonus against them. In vanilla Civ though there's really nothing you can do in the early game except take coin-flip battles and hope they're stupid enough to attack units on defensive terrain. Not fun.
The "Final Frontier" scenario in Civ IV: Beyond the Sword has this with Space Pirates, which constantly ruin your day by using your own hyperspace tunnels (i.e. space roads) to reach your systems and destroy your construction ships (i.e. workers). However, since there are no other "terrain" improvements in the scenario, that's all they can do.
Xenon "M" interceptors in X3: Terran Conflict. While they're poorly shielded, they come in huge swarms, and 90% of them will mount Pulsed Beam Emitters — rapid-fire, lethally accurate lasers that do incredible damage to shields. A single M can strip down the shields of an M3 fighter very quickly on its own — add in several dozen friends and you basically need a capital ship to kill them all without dying instantly in a hailstorm of (nearly) Hit Scan weapons.
The Xenon "Q" frigate is a Demonic GiantSpider. On paper its stats are actually fairly mediocre for a frigate (though it typically mounts some pretty heavy weapons, including PPCs on the flank batteries), and the firing arcs for its turrets are totally bugged in in-sector combat. However, it is reviled for its ability to one-shot pretty much anything with less than ten gigajoules of shielding in out-of-sector combat. Technically every frigate can do that given a lucky break with the RNG, but as the Q is by far the most common enemy frigate it gets all the hate.
If you're in anything smaller than a corvette, Khaak Clusters. They're made by one M3 ship and a lot of M5s that travel as a single target, then separate and attack when an inviting target approaches. None of them are a threat by themselves, but if the cluster isn't distracted by other targets and chooses you, you and only you to give its attention to, then you'll be facing Death of a Thousand Cuts as 10+ ships concentrate Hit Scan weapons on your ass. If you're good enough of a pilot you can probably take the smaller Clusters in an M3, but one variety has 26 ships in it.
Many of the randomly encountered Pirate M3 and M4 ships tend to equip the Plasma Burst Generator. Think Video Game Flamethrowers Suck? Not here. A PBG-equipped Blastclaw can shred most anything smaller than a corvette once it gets into range, and is nearly impossible to escape from. Thankfully, once you get a good corvette like the Vidar or the Springblossom (or a frigate like the Panther or Shrike), they lose a lot of their menace.
The Dread Lords in Galactic Civilizations. Any conflict involving them is going to be bloody, to the point where their constructors can go through mid-level ships like knives through butter. Also, ten Dread Lords infantry can conquer a planet. We recommend either swarm tactics of a sting-before-you-die suicide attack nature, or blowing up their sun.
All these tactics are just delaying actions, as you, hopefully, race to research and build better, stronger ships. Eventually, you can even take them in a one-to-one fight, although it will be difficult getting there.
The native lifeforms in Sid Meierís Alpha Centauri, especially since their attacks completely ignore your units' armor, and the only way to beat them is to have high morale or a special psi-shield (which doesn't do much against enemy units). The xenofungus itself is extremely annoying, as it constantly spreads, destroying your terrain improvements. Even removing it completely from an area doesn't guarantee that, in a few turns, a random scripted event won't create a highly-concentrated xenofungus area complete with boils of mind worms and a spore tower or two.
Even the designers knew these things were irritating; notice how many of the construction quotes refer to the colonies's struggles with them.