in Magic: The Gathering, the Simic experiment "Project Kraj" was deliberately designed to break free and become uncontrollable, because that's just how the Simic guild does things.
In-game, execution errors or opponent interference can lead to certain combos repeating without end instead of ending in a useful result, which the rules treat as a drawn game. For example, three copies of Oblivion Ring or similar cards can juggle each other in and out of exile; their controller can break the cycle by exiling something else ... unless all other eligible targets are gone. This actually happened once to a top pro, with hilarious results: See here.
Decks based on card drawing combos can be very powerful, but if you aren't paying attention you can draw yourself completely out of cards—an instant loss.
They also created a bunch of psychic warrior races who could manipulate the Immaterium. Technically they did manage to put the enemy out of action, but only by accidentally infesting the galaxy with Enslaver parasites that wiped out virtually all intelligent life.
This also happened to the guys the Precursors created to fight. The Necrontyr allied with the C'tan to get the power to defeat the Old Ones. The C'tan gave it to them...by eating their souls and turning them into mindless killing machines. And then the C'Tan got hit with this when the Necrons who hadn't completely lost their personalities rose up and wasted them.
After the Thousand Sons legion turned to Chaos, mutation became so endemic that it threatened the army's survival. The head sorcerer, Ahriman, came up with a mighty spell to purify the unstable soldiers while enhancing the psykers' powers even further. He succeeded, to an extent—though the psykers were spared, the rest of the legion was reduced to a bit of dust and a spirit sealed inside their power armor, leaving the majority of the Thousand Sons as mindless automatons. Their primarch Magnus the Red exiled Ahriman for this failure, and was not in the least bit comforted when he was reminded that since the legion's patron was Tzeentch, this "success" may have been all according to plan.
During the first age of space colonization that came to be known as the Dark Age of Technology, the people who settled the ice planet that would one day become the Space Wolf homeworld Fenris were gene-spliced with arctic wolves to allow them to adapt to its unforgiving environment. Now the entire planet, apart from a few balmy southern islands, is infested with slavering, mindless mutant wolf creatures that are nearly impossible to kill.
A revelation into the Eldar backstory says that quite a few of the Eldar pleasure cults shortly before the Fall were actually deliberately engineering the necessary psychic resonance to create a Chaos god. They believed the new god, made from their feelings of "pleasure", would allow them to transcend mortal forms and live new immortal lives of eternal bliss. Didn't go as intended.
In M40, the Officio Assassinorum tried to create an ultimate assassin that wouldn't rely on fallible technology. They succeeded, but their creation, a being called Legienstrasse, escaped. Punitive forces were sent to destroy her, but she killed several dozen Space Marines (mostly first company veterans, Terminators and Epistolary), Culexus and Eversor Temple Grand Masters, and was finally taken down by Capt. Lysander of the Imperial Fists. Probably.
In the Horus Heresy novel Fear to Tread, dealing with 40K backstory, some daemons attempt to awaken the Red Thirst in all the Blood Angels simultaneously. The plan was to use it to lure them to Khorne. What it actually did was ensure the daemons' failure, because it turns out that in full berserk, the Blood Angels were capable of tearing through lesser daemons like tissue paper...
In Teenagers from Outer Space, getting a critical success on a skill roll is just as likely to have hilarious but unfortunate consequences as a critical failure. The rulebook gives the example of flirting with a girl causing her to fall madly in love and become a Clingy Jealous Girl.
Paranoia has disastrous critical successes, mostly on mutant power rolls but potentially on anything else the GM thinks would be amusing. It's especially vivid and spectacular when the power is something like pyrokinesis.
Changeling: The Lost has a Dangerous Forbidden Technique in the form of a goblin contract that calls the Wild Hunt. If you succeed, they will show up in ten minutes, which gives you time to make your get away. If you critically succeed though, they show up next round. The game also provides several different options for calling up one of the True Fae, any of which is almost guaranteed to end in this trope.
A pretty common occurrence in Mage: The Awakening, especially when you are using fate magic. Honestly, given how most of the downsides of the use of arcane power are indirect (your enemies can find you, spirits take notice of you, things going right tempts you to be just a little more ambitious next time...) and how fond the setting is of the law of unintended consequences (several of the other splats have potential origin stories starting with "one day a bored mage thought he'd try..."), "gone horribly right" is essentially the unofficial slogan of the series. It gets more and more direct and prominent depending on the creativity of your ST, especially if you give him the opportunity to play with Exact Words by interacting with spirits, the dead, compulsions, the fae....
Mage: The Ascension had the "Arcane" advantage, which was the single most powerful tool for preserving The Masquerade and staying alive in the World of Darkness. The quality meant that you were 'loosened' from the fabric of society— at lower levels, paperwork and photos of you tended to get lost after a few weeks, and at the highest level any non-mage would forget you existed within moments of looking away from you. This was a huge lifesaver 90% of the time, made utterly foolproof by the fact that it was 'always on'. The other 10% of the time, well, it was always on. Hope you didn't need to rent an apartment. Or call the cops. Or call an ambulance.
In fact, this is actually a backstory element for the game as a whole; back in the Medieval period, a Tradition of Science-Mages called the Order of Reason decided they had to take over the Reality Consensus for the sake of humanity, which suffered due to the almighty power of the Sorcerous Overlords of the other Traditions. It worked, making Science the truth and Magic a fiction. What went wrong? Well, two things, really. Firstly, they ended up stifling human imagination in general, causing their own "hyperscience" to become as unreliable as conventional magic, and making it increasingly difficult to recruit people. Secondly, they ended up sliding down their own slippery slope, starting to become almost as bad — and sometimes worse — than the Traditions they overthrew.
Exalted: as one of the Freelancers put it, the process of creating the Solar Exalted required a being without limits to push himself to the brink. The weaponized humanity was then fielded against the creators of Existence, and created a hegemony of power that Heaven couldn't have destroyed if it wanted to. The hegemony only collapsed because the Exalted ended up turning on each other.
It sometimes happens for players to deck out and take an instant loss for going too fast. Lightsworns are a prime example, as they need to discard cards from the top of the Deck to activate their effects, which usually is good, but can quickly create a deck out if you're not careful. Another hilarious example is Infernities, which can easily make an opponent's activated Maxx "C" note A card that allows you to draw cards whenever an opponent Special Summons, which doesn't stops your opponent's plays but usually gives you so much advantage most players just stop Special Summoning on the spot turn into an one-turn kill with the right setup just because they can Special Summon that much.
In 3rd Edition Dungeons & Dragons, legend has it the Keepers were created (or discovered) by a wizard to collect all the knowledge of the world. Then he grew paranoid, and ordered them to ensure no-one else had this knowledge by destroying its potential sources. Naturally, they immediately killed the only non-Keeper with access to the collection.
In the specific campaign, The Elfslayer Chronicles, the DM played it so that elves were perfect, and that humans were bigoted, overly-patriotic warmongers. One player got fed up with this, and started playing the game in exactly the way it would be sensible to go, and exactly the way the DM didn't want. More specifically, the DM wanted to have her players stop an ongoing war between the elves and the humans, started when the elves supposedly killed a human prince, by taking him back to the human kingdoms and showing them that he was alive. But, the war would actually be better for the human nations than simply trading with the elves. So, he had his human illusion mage character kill the prince and frame an elf for the crime.
Another specific campaign: That Guy Destroys All Psions. The campaign was at level 20, and the author wanted to make a rogue with an odd, if effective magic-using build. The DM vetoed this and told him, essentially, to "minmax or GTFO", which rather pissed him off and led to him creating an uber-optimized wizard that left the plot in shambles.
Erandis d'Vol of Eberron is the result of a project that went horribly right for the participants. At the time, the elves and dragons were in a war. Erandis' family, House Vol, secretly collaborated with a green dragon in a bid to end the war, assuming that by producing a half-dragon and therefore showing the two races could be united, it would encourage an end to the fighting. That half-dragon was Erandis, and when she was announced to the world the war did end... because the vast majority of both elves and dragons were so outraged by this that they turned on and jointly obliterated House Vol. Erandis is the sole survivor or descendant of House Vol proper left alive.
A built-in one from the "Reign of Winter" adventure path for Pathfinder: in the second adventure, the easiest way for the party to get into the city of Howlings is for one human character to wear a magical cape, known as a rimepelt, that lets them pass as one of the winterwolves that rule in the city and be diplomatic with the guards. However, one of those guards is a female, named Greta, who is single and looking for a mate. Completely fooled by the player's act, she thinks they're a real winter wolf and starts flirting with them. Fortunately for the party, the recognition of her potential Abhorrent Admirer status (since she's Neutral Evil and, y'know, a giant wolf) was obvious and thusly she'll back down quite amiably if they're diplomatic about refusing her advances. Be a Jerk Ass about it, however, and she will get very ticked off. Greta's flirtations, in a game where the player responds favorably, could lead to this happening from her viewpoint; she ends up dating (and possibly more) with a human rather than another wolf, she decides being different species doesn't matter, and may even undergo an alignment change to try and make her lover happy.
Eclipse Phase: an experiment to create AI that could self-improve without limits led to the creation of the TITANs, which ended quite badly for everyone involved and several billion people who weren't Also, gaining large amounts of reputation with certain factions, depending on location, or criminals anywhere. You need it to obtain gear, but the side effect of approaching 100 rep is that everyone knows you're a big-shot in the criminal underworld or whatever. Cover IDs and other methods of identity concealment being extremely weak compared to most systems exacerbates this a lot.
The setting of Kyuden Kurogane-Hana in Legend of the Five Rings has this as part of its backstory. Two lords both wanted control over a particular valley containing a shrine to the Fortune of Death. Lord Seto, the lord who currently controlled the valley, sent his courtiers to the Emperor's Court to try to prevent his rival from taking the valley through political methods, while Seto built a powerful fortress to prevent the valley from being taken by force. The courtiers, meanwhile, managed to befriend the Heir to the Throne and told him all about the wonderful shrine and their Lord's good stewardship of it. They were so successful at impressing the Heir that when the Old Emperor died and the Heir assumed the throne, he announced that his father's funeral would be held at the shrine in Seto's lands. The good news: the fact that the shrine was going to be the burial site of an Emperor made it a sacred site and immune from any military or political attack. The bad news: Seto now had only a few months to take what was supposed to be military fortress and turn it into a palace capable of hosting the entire imperial court...
That's not all. The new Emperor (and court) is going to be paying his respects to his father regularly, as the anniversary of the death, birth and other significant events of the old Emperor is pretty important. A seige by hostile forces would likely have been easier to repel and cheaper to endure.
In Fiasco, one of the Tilt elements is "A stupid plan, executed to perfection".