In Project "Ma", Adam Moonlit (played by KAITO) gave Eve (played by Hatsune Miku) a drug, which made her miscarried her twins. It not only ruins his chances at greatness but also makes him realize he's hurt Eve.
Relient K's "Deathbed" lampshades this. The narrator is on his deathbed and recounting his life and all the mistakes he made. Including a shot-gun wedding, a love-less marriage that ended in divorce, a few kids that are implied to not see him much after said divorce, and a drinking problem that nearly kills him at one point. At the end Jesus appears to take the narrator to heaven because he repented heavily in his last few years.
You cried "wolf" / the tears they soaked your fur / the blood dripped from your fangs. / You said "What have I done?"
Reese Roper: You may notice that we are not rock stars, because you will hear these mess-ups and you will say to yourself, "My God, what have I done? What have I done?!" Oh, yes, you will.
The House of Heroes song "Voices" is about a soldier having this kind of realization.
In the silent hour I can hear them/ Pray to the Mother but the Mother doesn't love my soul/ In the blacked earth lay my secrets/ The hounds of Hell know everything...
"The Ballad Of You Know Who" by Richard Swift deserves mention, for using this phrase as the entire chorus of the song.
"Once In A Lifetime" by Talking Heads quotes the trope-naming phrase verbatim.
Iron Maiden's "Killers" contains the line "Oh god help me what have I done?". As the song points out "his blood lust defies all his needs", meaning that killing people is what turns him on. Even though he regrets it, he can't stop doing it.
Iron Savior, of (unsurprisingly,) Iron Savior has one of these, touched on in a few songs. "I've Been To Hell";
The crowd murmurs this a couple of times in "Hope Rides Alone", although it seems to boil down to an Ignored Epiphany.
The first line of "It Gets Better" by Fun.: "What have we done? Oh, my God!" The answer? They lost their virginity together.
Similar to the "Handlebars" mention above: Zero Sum, the final song on Nine Inch Nails' album "Year Zero" is basically one last humble apology from humanity to everyone who was hurt by the dystopian government that was allowed to come into power in the first place. All while the world quietly ends in the background.
Shame on us
For all we have done
And all we ever were
Just zeroes and ones...
Criminally Insane by Slayer.
Disapprobation, but what have I done
I have yet only just begun
To take your fuckin' lives!
"Evaporated" by Ben Folds Five uses this line as the last line of the chorus - though it's never actually revealed what the singer actually has done.
Forgive Durden's musical, Razia's Shadow uses this trope in a near-verbatim manner at the conclusion of "Toba the Tura," when Ahrima comes to grips with his sin of destroying the lamps.
What have I done?
Please make me your son
What have I become?
Destroyed all I love!
Also in Razia's Shadow, Pallis' reaction to stabbing Adakias, thus causing his brother's death. While having been aware of what he intended to do, his reaction is full of regret; "Brother, what have I done? My blade has pierced your side. This was never my intent, oh god, please stay alive!" And then "Please don't let your tired heart stop beating. You're bleeding. Just keep breathing!"
In the song "The Flame" by Chimera, the lasts lines are "Oh God…What Have I Done"
The protagonist of Genesis' "One for the Vine" becomes the murderous conquerer he (as a lowly foot soldier) had deserted at the beginning of the song. No, not just a very similar figure - the exact same one.
The Iced Earth song "Gettysburg(1863)-High Water Mark" features an example in the form of Robert E. Lee lamenting the Confederate army's loss at the battle, due to his plan.
I look across this blood soaked land
All this blood is on my hands
God forgive me, please forgive me
It's all my fault, the blood is on my hands!
In the song "In the Glass" by OK Go, the protagonist immediately regrets his decision to become his reflection.
Though you'll never hear these words in Pink Floyd's The Wall, the songs "Hey You" and "Stop" serve this purpose. In the first instance, he realizes exactly what he's done by completing the wall; in the second, he's horrified by what he's turned into.
The Final Cut also features this line: "What have we done? Maggie [Thatcher], what have we done? What have we done to England?"
The third refrain from "She's Leaving Home" by The Beatles:
She (What did we do that was wrong?)
Is having (We didn't know it was wrong)
Fun (Fun is the one thing that money can't buy)
Machine Head's "Now I Lay Thee Down."
What have I done? I've gone and killed the only one I love. How could I do this?
The title track to Underoath's debut album Act of Depression—whose subject matter differs drastically from what they would do later—combines this with a Perspective Flip. The first two thirds of the song accounts the thoughts of a depressed person Driven to Suicide, who makes clear that being bullied was the deciding factor in finalizing the choice. The song then switches to the POV of one of the bullies, who has this very reaction upon seeing the dead body of the song's initial subject.
Victim: Thanks to all the people who drove me to death. Without you, I could have never ended my breath. Through your anger and hate, I was able to choose my fate. There was a way out, but I chose the easy route.
Narrator: Ice cold fingers, body lays on the floor. Pool of blood you see, you scream out in terror. Her body is now a part of mutilation, her soul the victim of strangulation.