Demonic Spiders: Arquebus enemies in a melee. Unlike archers, whose arrows travel slowly enough to dodge, bullets travel very quickly. And since one hit will slow you to a crawl, you'll usually suffer a second, fatal hit right afterwards.
Goddamned Bats: Any rival with rock-bottom honor. They have nothing to lose, and so they will endlessly try to assassinate you and kidnap your family. This means that if you stay home, you'll have to do lots of swordfights, and if you leave home, you'll lose a family member and end up doing some Melees to get them back. It's not that they're terribly difficult to defeat, but it gets annoying very quickly.