Spinners can be quite straining on the player's hand and unprotected touchscreens. Easier dificulties at least put spinners at the end of segments, giving you a bit of cooldown time, but harder difficulties often make you go directly back into notes immediately after the spin.
The scoring system, which adds a multiplier that is proportional to your current combo. This has the side effect of combo breaks in the middle of a song being much more damaging than a combo break at the begining or the end. Just to be even weirder, grading is calculated separately from score and is based on judgement counts, but score is prioritized over rank for score-recording purposes, meaning that you can lose your S-rank to a higher-scoring A-rank.
The lifebar drains constantly. Very few other rhythm games can fail the player in the middle of a combo. This also means that if there is a lull in notes and your lifebar is in the "!!" section, you're dead (i.e. a difficult section is actually safer than an easy section if your bar's almost out).
That One Level: The final songs in both games are enough to make you break your DS.
The final segment of "Shanghai Honey" on Insane has eighth-notes suddenly spread out and spaced like quarter-notes, and is nothing like the segment on Hard.
The last 14 notes in "Countdown" on Hard and Insane are near-impossible to get all 300s on, especially when the song is so fast-paced.
"Believe" from Ouendan 2 on Expert. The song is easily the slowest in the game, even slower than "Over the Distance" from the original. Unfortunately, your life gauge still drains extremely quickly. This means that to make up for the life loss between notes, you need to get practically every note perfect to make it through. Some funny timing doesn't help matters.
And to hammer just how hard it is to keep your life meter up, it's entirely possible to fail all the cutscenes with an S rank.
"Neraiuchi" (a.k.a. the "Cleofatra" stage), not so much because it's difficult in and of itself, but because it has a whopping six spinners, distributed in just the right places to give you hand cramps for weeks. To put it in perspective, the only other song in the game that comes even close to that is "Ready Steady Go", and being the final song, it's supposed to test your limits. This, however, comes right in the middle, which makes most of what comes after a giant Breather Level in comparison.