The Detective ConanNon-Serial MoviePhantom of Baker Street has this as the main premise. The VR game console was hacked by an AI called Noah's Ark, and the 50 players have to Win to Exit... or their brains will be literally fried.
To Aru Kagaku no Railgun has the cast modeling swimsuits inside a holodeck. Naturally, it malfunctions, and they start getting transported to random "locations," culminating in a 2001homage on the surface of the moon.
The X-Men's Danger Room goes haywire almost as often as Star Trek's holodecks. While it's usually because some enemy has intentionally tampered with it rather than a random malfunction or user error, it did eventually develop sentience (or rather, turn out to have been sentient all along) and decide to kill the X-Men.
Westworld features the killer robot variant of this plot. As does its sequel Futureworld.
Used briefly in the MST3K-worthy Overdrawn at the Memory Bank. Aram Fingal gets "doppled" into a baboon in an Africa simulation; everything goes south when he gets attacked by an elephant. The technicians are able to pull him out of the simulator quickly enough, but can't find his body to plug his mind back into.
A virtual reality adventure gone out of control provides the basis of the plot of Dennis L. McKiernan's novel Caverns of Socrates.
The Ray Bradbury story The Veldt featured an educational holodeck program about the animals of the African plains. When the kids begin bypassing the safety protocols, their parents get worried about how real the simulation seems and try to shut the simulator down. When they try to retrieve the kids, the kids send the lions to eat their parents, so they can stay with the animals forever.
In Revelation Space by Alastair Reynolds, deliberate sabotage turns a training sim into a deadly trap.
Vivian Vande Velde's Heir Apparent and User Unfriendly both deal with virtual reality games gone wrong; both games, ironically, were made by Rasmussem Enterprises. In Heir Apparent, people protesting the violence in Rasmussem's fantasy games (and fantasy in general) damage the computer equipment to which the protagonist is connected, forcing her to either win the game or die. in User Unfriendly, the protagonists have gotten a hold of an illegal copy of another of Rasmussem's games rather than pay for time, and are playing it at their home. That doesn't go as planned either, and again, the only way out is to win.
In Piers Anthony's Killobyte, the titular VR game is full-body immersion. A hacker calling himself Phreak makes a virus that keeps people from exiting normally, forcing them to wait until someone can manually pull them out, crashing their character and making them start over. While normally a harmless prank (he enjoys taunting his victims and messing up their games while they're locked in), this turns into a life-threatening situation when he traps a diabetic and a man with a pacemaker that could be shocked into malfunctioning by the game's death-penalty.
The Nightmare Machine from Galaxy of Fear. It's actually intended to take visitors through their worst fears, one by one. The Arrandas try it out, don't like it, leave, and go about their day, but later find they're actually still inside and can't just cut the simulation. Actually it is a psychic monster and they were intentionally trapped by order of the Big Bad.
In The Star Kings by Edmond Hamilton the League's secret weapon used similar principle. During a battle Cloudmen would tap "telestereo" beams and insert recordings of shooting weapons. The energy output would be enough to destroy everything on the bridge within sight of a receiver, putting the ship out of battle or making it a sitting duck. Looks like telestereo receivers were built unreasonably powerful. Fortunately, all ships already had countermeasures — portable dampers that can suppress the shots leaving the receiver pad — and started using them when the hero figured out how the weapon worked. Probably the League was not the first to invent those.
The most common "simple" breakdown is to lock the senior officers inside and turn off the safety protocols. More extravagant scenarios can occur, such as poorly-worded instructions resulting in a fully sentient simulation of Professor Moriarty gaining complete control of the Enterprises's computer. The tendency for the holodeck to malfunction like this has become rather infamous.
At least three times their holographic technology has accidentally created fully actualized sentient beings. Though in each instance, there was a deliberate action taken that simply had an unintended consequence. Which is really rather ironic when one considers how much significance is attached to Data (and his sibling Lore) being fully-sentient androids, a technology which apparently only their creator fully-understood and which the Federation cannot duplicate (at one point they wanted to disassemble Data to try to figure it out). Yet it would seem that starship computers can generate fully-sentient holograms with just a poorly-worded command from a user.
Voyager played with this in one rather trippy episode, which starts off with the Doctor embroiled in an apparently "mundane" crisis, only for increasingly weird things to happen. Eventually, he's told that the whole thing is a Holodeck Malfunction. It is, but not the one he's being told it is.
One time they left a program running too long and eventually the perceptual filters (which apparently keeps them holograms from noticing anything that doesn't fit the parameters of their program) futzed out and they became aware something weird was going on. The crew fixed that up by letting the holograms believe they were real, but the crew were time-travelers.
Lampshaded when Worf joined the crew of DS9, and reminisced about his time on the Enterprise with an old shipmate:
Worf: We were like warriors from the ancient sagas. There was nothing we could not do.
Though there was that one time everyone's brains got transferred into Bashir's James BondExpy Holonovel, and the safeties were turned off. Bashir at one point shoots Garak to stop him from leaving the holosuite thus potentially melting everyone's brains. In this case the holodeck malfunction actually saved there lives as without everyone's brains would have simply ceased to be.
Averted — or maybe inverted — in the Star Trek: Deep Space Nine episode Badda-Bing, Badda-Bang. The holographic Vic Fontaine is threatened by mobsters, and if the crew doesn't save him, he'll be permanently deleted. Deleting the bad guys or reprogramming the holodeck simply won't work: they have to solve the problem in-game and in-character. What's notable is that it's NOT a malfunction: it was programmed into the scenario. (The programmer never dreamed that the crew would consider losing Vic to be the death of a good friend.) This is likely the only time in Star Trek that a crisis is caused by the holodeck operating exactly as intended.
A large part of the premise of Virtuality (the failed 2009 pilot, not the film).
Red Dwarf: The simulation program in "Gunmen of the Apocalypse".
A subversion involves one of the game characters coming to life and Rimmer ends up shooting him with a bazookoid, helping to gain the confidence to become Ace Rimmer. Turned out, Lister had dressed up to play the part and loaded the gun with blanks.
The "Better Than Life" simulation in the books inverts this by working exactly as designed - it really is 'better than life'. Trouble is, once you're in, you aren't aware it's a game and even if you are, it's so good you don't want to leave. End result: you starve to death. The Dwarfers exit only by Rimmer being such a twisted and bitter human being that his neurosis first turns his own fantasy, then the others', into hell.
In Power Rangers in Space, lightning somehow results in simulated monsters breaking free, going to Earth, and impersonating townspeople to lie in wait for the Rangers.
The X-Files episode "First Person Shooter" featured a video game designer's fantasy wish-fulfillment character gaining sentience and infiltrating another designer's prototype virtual realityFirst-Person Shooter game to kill players (who, of course, die in real life). A famous gamer is brought in (but fares no better) and ultimately Mulder and Scully end up going in to take down the marauding avatar. Rather than, you know, just scrapping the killer video game or loading a backup copy of the game onto a different mainframe or something.
In Warehouse 13, a prototype game console traps the users inside, and started to use their own fears against them. One of these fears grows strong enough to steal the controllers from the players, effectively trapping them inside.
This was the plot of the X-Men game for the Sega Genesis. Magneto infects the Danger Room computer with a virus to turn its simulations deadly; every level except for the last one involves beating these simulations.
The game had an relatively subtle reference to Star Trek: The Next Generation, in that when the simulation ended, the simulation room looked exactly like the TNG holodeck.
Star Soldier: Vanishing Earth's first stage is a training exercise that gets hijacked by the Big Bad near the end.
Optic Sunflower's stage in Mega Man X 8. The stage is a Maverick Hunter training base that makes heavy use of VR; to progress safely, you have to pass the "tests" with flying colors.
The "Operation: Anchorage" DLC for Fallout 3 takes place in a VR simulation of General Chase's insanity-scrambled idea of what happened during the US Army's counterattack to drive the Chinese out of Alaska in the Great War. If you die in the sim, you die for real, but subverted in that you're warned ahead of time that the simulator's safeties aren't working.
Gunnerkrigg Court parodies this: Dr Disaster sees several students and a teacher disappear from his simulator, and immediately assumes that the sim has trapped them. (Unknown to him, they had merely teleported to another building.)
Lampshaded in Intragalactic. When the cast gets their spacecraft repaired, the mechanic points out their holodeck is an unsafe model, prone to malfunctioning, to which captain Benjamin replies that it's the whole point of it. Holodecks just aren't fun unless they periodically lock people inside and turn off safety protocols.
Bob and George has Proto Man lampshade this during X's introduction. The second time the holodeck is used, it breaks while Proto Man is attempting a convoluted plot to get X's buster and holodecks even gets referenced by another character at one point.
In one story in the Whateley Universe, two hackers went after Team Kimba by trapping them in a sim without their armour or weapons, facing a group of pissed-off simulated attackers. The Kimbas manage to use their smarts to get out, but it's a Pyrrhic Victory.
The Futurama episode "Kif Gets Knocked Up a Notch" parodied Star Trek with the Nimbus's Holo-Shed, which malfunctions and causes History's Greatest Villains to come back to life: Attila the Hun, Professor Moriarty, Jack the Ripper, and Evil Lincoln. Judging by the other characters' reactions, this sort of thing is distressingly common.
The Itchy-And-Scratchy Land episode of The Simpsons (a parody of Westworld).
Star Trek The Animated Series gives the original NCC-1701 Enterprise's rec room a hologram feature, a full decade and then some before TNG's (in)famous holodeck. You get no points for guessing what happens.
The episode "Once Upon a Planet" featured the crew returning to the amusement park planet of "Shore Leave" (see Live Action TV) to find that it was now actually hostile.
Happens twice in Winx Club, once in the first season and once in the second. The Trix had sabotaged the simulator both times.
normally only dangerous if they lose, and limited to the area caught in the game
. When Megabyte extracts Mainframe's core energy from the Principal Office, a Game Cube that lands on it gets corrupted, blending aspects of the game reality and the Principal Office. If the User wins the Principal Office gets destroyed and Mainframe crashes, but if anyone else wins the core energy leaves with the game and Mainframe crashes. Bob has to keep the game running until he can get the core energy back inside the Principal Office to stabilize the game and let it leave safely.
Young Justice has the episode '"Failsafe," in which Miss Martian, unable to process that she is in a training simulation after watching Artemis "die" in front of her, loses control of her powers and rewrites the entire team's memories so they believe the simulation is real - including their own deaths, causing them to slip into comas in reality.