The planet Miranda in Serenity is a Shout-Out to William Shakespeare's Tempest and by extension, Huxley's Brave New World. Of course, this is also a reference to real-world naming conventions, as all the moons of Uranus are named after Shakespearean characters, and one moon is in fact named Miranda.
Alan Tudyk's character in V (2009) was impaled by a stake... Sound familiar?
In the pilot, when the crybaby is released, Wash gives a shout out to The Beatles song of that name.
In "Out of Gas", we learn that Serenity's first mechanic was a guy named "Bester." Later we hear Mal call him "Alfred." Alfred Bester was an important author in the post-war, founding era of modern sci-fi (as well as a character in Babylon 5).
In a deleted scene from the pilot episode, Simon's encyclopedia can be heard mentioning a General Richard Wilkins who led the Alliance forces at the Battle of Serenity. Mayor Richard Wilkins was the Big Bad of season 3 of Buffy the Vampire Slayer, also created by Joss Whedon.
"I was aiming for his head." - Avon says exactly the same line with regards to Travis in Blake's 7.
The Weyland-Yutani logo is visible on the heads-up display of the anti-aircraft gun used by Mal in the pilot.
Jayne's line "I was gonna get me an ear too" is very likely a Shout Out to Reservoir Dogs.
One of the fictional historical figures is called Shan-Yu.
Jubal Early's line "Well, here I am" could be a reference to the song "Space Oddity" by David Bowie: "For here am I, floating round my tin canů"
You might be aware that Joss Whedon is a vivid reader of Henrik Ibsen. Mal being left naked in the middle of a desert is a Shout-Out to Peer Gynt, who ended up being robbed and quite naked in the middle of the Sahara wasteland at one point in the play. By a bedouin girl who seemingly fell for his act.