Anti-Climax Boss: Long John Peter. His level can be very annoying to some players, especially due to the wave upon wave of pirates you have to fight on a single health bar. Long John Peter himself, however, just has a few rather predictable patterns and is pretty easily beaten.
Crosses the Line Twice: Being Family Guy, this was inevitable. Whole levels count as this, such as the Amish, Handicapped, and Santa's Sweatshop levels. Not to mention assassinating Mayor McCheese.
Demonic Spiders: The explosive dolls in the North Pole level. They move very quickly, and their explosions deal a lot of damage. It's almost impossible to kill them before they hit you.
The muscular Bertram clones in the final level are incredibly durable and do a lot of damage. They usually seem to attack in packs and can kill you in a matter of seconds if you're not careful.
Goddamn Bats: The large wheelchair enemies with tasers in the handicap universe level can ram you, which stuns you for several seconds and leaves you open for attacks by other enemies. They often do this from offscreen when you aren't paying attention to them, or several may do this to you in sequence. This is especially annoying during the Crippletron boss fight. If there are three or more of them, you have to be especially careful because they can trap you in corners and completely drain your health in seconds with their tasers.
That One Boss: Evil Stewie. His tank is pretty much a Damage Sponge and the blasts from the cannon can do immense damage depending on how he hits you. To make matters worse, Evil Mayor West flies around on a jetpack throwing bombs at you (which take away about a quarter of your health) during the fight.
That One Level: Many players dislike "Pussy Whipped" (the evil universe level).