The fantasy Ringo has in Episode 4 has Tabuki sing 'Dazzling the stage!' as he made his entrance, just like Takuto in Star Driver. This becomes Hilarious in Hindsight when you realize Tabuki's voice actor is Akira Ishida, who voiced Head in Star Driver, the antagonist and Takuto's father.
In another fantasy, Ringo talks to a tree who looks suspiciously like Wispy Woods from Kirby.
In episode 5, the painting hanging behind Ringo in the restaurant is definitely a shout out to The Kiss by Gustav Klimt.
The whole show is filled with references to Night on the Galactic Railroad: the conversation between two boys in the the first episode (and the last), apples, trains, and the sacrificial scorpion... much of Penguindrum's symbolism builds off of this book and movie.
During the "Rock Over Japan" sequence, Himari/Princess of the Crystal strikes a pose◊ identical to one used by Usagi in Sailor Moon. Note that Ikuhara started out directing Sailor Moon.
When we see the Goddess of Sanetoshi's story in episode 13, her body is wrapped in glowing red ribbons the same way Usagi is during the transformation sequence, especially in the Sailor Moon R movie, which Ikuhara directed.
The scene where the possessed Himari places her hand on Kanba is a near shot-for-shot remake of the sword pulling scene from Revolutionary Girl Utena. Not surprising given the director, again.
Two characters from the opening bear resemblances to Juri and Utena (right down to the black and white color scheme from her movie clothes).
Also regarding Utena the scene in the opening where the lyrics mention the end of the world is similar to the Student Council elevator stock scene. The scenes that followed involved the council members discussing End of the World's plans.
The scene in the hospital in ep 5 with Masako stroking Kuho's cheek is nearly an exact remake of Juri stroking Utena's cheek in ep 7. Then it becomes Mood Whiplash when Masako erases Kuho's memories, similar to how the scene in Utena ended with Juri lashing out at Utena.
And the scene where Masako fires her memory erasing balls at the two women and they explode resembles the scene of Anthy being impaled by the swords of hate in terms of composition and color scheme.
Speaking of which, Yuri's costar, Tsubasa, is a dead-ringer for a certain cross-dressing swordswoman...
In general, the all-female acting troupe Yuri belongs to is a very obvious Serial Numbers Filed Off version of the Takarazuka Revue. Her affair with one of her costars is likely a reference to the lesbian overtones a lot of people have claimed are inherent to the Takarazuka concept.
The boys' school uniforms seem to be modelled after the one worn by Utena, albeit with long trousers.
Most of the family names in the show are borrowed from the actors in the movie Nankyoku Monogatari, about a 1958 Japanese scientific expedition to the South Pole.
The name of one of Sanetoshi's two rabbits, Souya, was the name of the icebreaker used on the same expedition. The other rabbit, Shirase, is named after another icebreaker that was also used on Antarctic expeditions.
The music in the musicbox in episode 10 is the New World Symphony by Antonín Dvořák.
And the scene where Natsume watches Kanba with night vision goggles while he can't see a thing in the dark references the climactic scene of Silence of the Lambs.