The lead-in to the theme tune in the episode "25", which was helped by brilliant cinematography. Bartlet finds out that Zoey has been kidnapped and he immediately drops the photos of her he's been sorting and his glass of scotch. He then glances at Abbey, and as the background music crescendos, the camera zooms in on one of the pictures. The entire sequence takes twenty seconds, but it's a brilliant 20 seconds.
The use of Massive Attack's "Angel" at the end of "Commencement". It's the perfect balance of creepy, tense and trippy as Zoey realises she's been drugged and then her Secret Service detail realise she's missing, and the repetition of "Love you, love you, love you" as Amy questions Donna's feelings for Josh.
The scene in "Noel" where Stanley pushes Josh to tell him how he really cut his hand and it's shown that music is making him relive the shooting. It's an amazing scene, if more than a little heartbreaking, and the music that helps make it so great?
Not to mention that Yo-Yo Ma insisted on playing the music live for every take rather than use a recording.
Also in "Noel": when Donna and Josh walk out of the White House past a group of carolers performing "The Carol of the Bells" at the end of the episode, the sound of sirens begins to rise as they finish the song... calling back to Josh's diagnosis and the manifestation of his PTSD.
Two amazingly fabulous pieces, set back-to-back and interlaced with scenes from a play the President is attending, in "Posse Comitatus" (season 3 finale):
"Halleluja", sung by the late great Jeff Buckley over the death of Secret Service Agent Simon Donovan.
"England Arise", by the play's actors intercut with the assassination of the terrorist (but Defense Minister of Qumar) Shareef by British forces.