Breather Boss: While each previous bosses all required a specific strategy to beat them, the Panda King is nothing short of a walk in the park. His attacks are all slow, blatantly telegraphed (not that that's a bad thing) and easy to avoid. And what do you have to do to beat him? Avoid his attacks while approaching him, then whale away on him with your cane. No other boss had you do this. At all.
Clockwerk definitely crossed it when he had the Fiendish Five kill Sly's father, but it turns out that his whole reason for it? He hates the Cooper line and wants to surpass their reputation.
While we only learn of the villain's main evil deeds in their stages introductory stages, we actually see the Panda King destroy a mountain village because they wouldn't pay his protection racket... then, when you enter his compound, you hear an announcement from the Panda King telling his workers to murder any survivors in case they come to the compound looking for vengeance. This, and his part in killing Sly's parents, is part of the reason why Sly refuses recruit him in the third game or forgive him.
Most Annoying Sound: Unlike Sly's smooth, quiet voice, his teammates yell their lines. Bently in particular is bad about this, given how often he speaks.
Paranoia Fuel: Ms Ruby is the only one besides Clockwerk to know precisely who is messing around in her base of operations.
Doesn't help that in the PS3 Collection the rhythm that you have to press the buttons in is off beat from the song. It can really annoy some of the more musically inclined players.
Muggshot as well, mostly because you have to have amazing dodging skills in order to dodge his bullets, and Muggshot is a pretty fast mover. To make matters worse, the second floor is oddly shaped with a few pitfalls and the last phase of the boss has you spire jumping to turn the mirrors around. Oh, and if you use a mirror as cover that's been turned to face him, it'll turn away when he goes to shoot you.
"A Ghastly Voyage", in large part due to the dozens of obstacles you have to shoot through to advance, not to mention all the Goddamned Bats.
"Sinking Peril". Remember that Daredevil Comet mission from Super Mario Galaxy where you have to rush through Melty Molten Galaxy without getting hit? Yeah, it's like that, but with a lot more climbing and precision jumping required. There are also portions of the level (electrified gates) that you can't rush through—you have to wait for the electricity to stop firing before you can pass.
"Bentley Comes Through," because every broken piece of code hurts you AND you have One-Hit Kill security drones after you as well. It gets to ridiculous Nintendo Hard levels like most of the final level.