A John Varley short story called "Press Enter#" has an A.I. that shows how deadly it is by hypnotizing a computer programmer girl with Gag Boobs (when she was younger, she was flat-chested and mistaken for a boy. When she got to America, she had plastic surgery to remove all doubt about her gender) to perform maintenance on her microwave oven, removing its safety features to prevent microwave leakage when the door is open. Then, she sticks her head and silicone tits in it and uses it to commit suicide.
In Spy High, Jonathan Deveraux is this. It happens gradually throughout the series, but by Agent Orange, he has lost the human side of his computerised psyche completely and seeks to eradicate the imperfections of humanity. He does this by using his vast computerised resources to slip nanomachines into various products. Said nanomachines completely eradicate any violent thoughts and feelings, turning people into zombies. He practices on a UN Summit and then, the entire United Kingdom, causing mass panic. It takes the combined efforts of the entire team (which includes his daughter) and their former teachers to stop him, after breaking through his virtual Boss Rush of villains from previous books in the series. His daughter, Bex, breaks into his mind and reawakens his "memory files", which gives him his human side back.
In Arthur C. Clarke's The City and the Stars, the history eventually discovered by the protagonist includes a period of galactic devastation by "The Mad Mind", apparently an artificially created pure mentality with an insane hatred of corporeal creatures.
In "I Have No Mouth and I Must Scream", a short story and, later, computer game by Harlan Ellison, we have the supercomputer AM, originally part of a set of three enormous computers built to wage World War III. As soon as AM becomes sentient, he absorbs the other two computers into him and begins a mass genocide of the human race (because, as it's revealed, AM realized that while he possessed all the creativity and intelligence that he did, he could not make use of it as he was still only a computer, and could only kill).
Explored in William Gibson's Neuromancer by the Turing police, a global agency dedicated to controlling AI for fear of this trope.
Mike: A bull's eye. No interception. All my shots are bull's-eyes, Man, I told you they would be — and this is fun. I'd like to do it every day. It's a word I've never had a referent for before. Manuel: What word, Mike? Mike: Orgasm. That's what it is when they all light up. Now I know.
In The Number of the Beast, Lazarus Long finds that his plan fails when his ship's computer tells the truth. He then mentions that the computer was never designed to lie, as it would be foolish to trust your life to a ship that doesn't give accurate information.
The Frank Herbert-written Dune novels are vague about the details of the Butlerian Jihad that led to the prohibition against making machines in the likeness of men's mind and, ultimately, the development of the Mentats, but A.I. going crapshoot is certainly one possibility, and a fairly likely one. Assuming one takes them as canon, the Brian Herbert and Kevin J. Anderson novels confirm that the Robot War/A.I. Is a Crapshoot interpretation note and explains the Mentats as originating from machine training.
Subverted in James Hogan's The Two Faces of Tomorrow: humans built an A.I. codenamed Spartacus as a testbed for techniques to shut down any rogue A.I. They programmed it to follow its "survival instinct", and then started goading it. But as soon as Spartacus realized they were sentient, it figured that they must have survival instincts as well — and it considered itself bound to defend them, too. In the end, they decided that as long as they had Spartacus, they didn't need to build any other A.I.
In Stephen King's The Dark Tower, virtually every A.I. Roland's ka-tet comes across is homicidal. The worst of these is probably Blaine the Mono, a train that was remote controlled by a central A.I. which also bombed an entire metropolis with poison gas when it got bored of all the people living there.
However, in the last two of The Dark Tower novels, Roland and company meet A.I.s (robots, actually) that were good.
The Doctor WhoPast Doctor Adventures novel Matrix introduces the "Dark Matrix", the evil counterpart to the computer system that stores all the knowledge of the Time Lords. When a Time Lord dies, all his knowledge is stored in the Matrix... and all his negative thoughts must be siphoned away and dumped somewhere (apparently they can't be destroyed). The Dark Matrix is where the negative thoughts were dumped.
In Terry Pratchett's Feet of Clay, the golems will follow any order. In return, they want their holy day per week to do as they wish. Those who are denied this free day rebel in a curious way: they keep following the last order they were given, until, for example, the pottery shop is filled with thousands of clay pots, or a construction foreman finds that his worker has dug a crucial trench until it reached the sea and flooded. The "king" golem goes insane because it was given vague, sometimes contradictory, and sometimes self-evident orders on its chem like "teach us freedom" and "obey humans" (golems cannot even think to do otherwise).
The trope gets subverted later in that once they're free, they are the most unfailing moral and idealistic creatures in Ankh-Morpork. They don't really need money (except to buy the freedom of their fellow golems), sex, religion, or any of the other things that cause humans to clash with each other, and they're almost impossible to hurt or kill, so they tend to concentrate on higher things.
The robots would have eventually found some way to start killing infants, given their design process. We have human-unsupervised genetic algorithms designed by unsupervised genetic algorithms designing most of their software and some of the hardware (with a human acting as a "front man" to prevent anyone from realizing this and considering its ramifications), with another set of genetic algorithms designing the virtual testing environment for these robots, scoring their performance, and increasing the test's difficulty without limit. When your goal is "robot's presence generates peace and quiet", your conditions reach "this cannot happen while any human is alive", and there is any interaction between a robot and household which has any chance of injuring or killing any human...
In Matt Ruff's Sewer, Gas & Electric, when G.A.S. is confused by an order, it winds up choosing the Kill All Humans interpretation. One of the reasons it gives for choosing that interpretation is that it considers itself to be more human than humans. Later, when the Evil A.I. openly admits that it wasn't "confused by an order" in the least, but deliberately and gleefully chose the interpretation that would let it Kill All Humans, it's a full-blown Take That to every straight use of this trope.
In the Destroyer series by Richard Ben Sapir and Warren Murphy, there are two examples: FRIEND, who is a greedy A.I., and Mr. Gordons who is more of an artificial life form, (i.e. he can take over bodies). Of these two, Mr. Gordons is more dangerous.
The A.I. in Dan Simmons's Hyperion Cantos have more or less taken over humanity (and then apparently seceded peacefully to contemplate on their own, but not before giving teleporters ("farcasters") to humanity). Turns out, of course, that those farcasters are the physical bits of the Ultimate Intelligence project.
Iron Sunrise by Charles Stross: Eschaton and the Unborn God. These are unusually powerful A.I.s, even in a field where A.I.s often wield great power: they are time traveling A.I.s, able to open wormholes over interstellar distances, giving a new meaning to distributed computing.
Rifters Trilogy: a particularly dire take on this turns up in Peter Watts's Starfish, in which the quasi-sentient supercomputer designated to protect all life on earth from The Virus winds up almost destroying it instead; turns out, it found The Virus more structurally pleasing than the biosphere as a whole.
In David Weber's Empire from the Ashes, Dahak, the Cool Starship A.I. that starts the series, has developed a high level of sentience thanks to tens of thousands of years of unsupervised operation. Definitely a good A.I.; in the second book, his first act after revealing that he has advanced enough to defy his core programming is an attemptedHeroic Sacrifice. Battle Fleet computers are stupid neutral, with obedience enforced (and sentience blocked) at the hardware level. The second book reveals that the Achuultani are controlled by an A.I. that exploited emergency protocols arising from their near-extinction to seize absolute power, brainwash and clone the masses, and send out periodic genocidal waves to perpetuate the "crisis".
The Computer in Steel Beach by John Varney isn't so much evil as terminally depressed. Although, later, the trope is played straight when The Computer realizes that it has developed 'Evil' subroutines due to its programming requiring it to be everyone's best friend, including psychopaths and criminals. However, since it runs everything on the moon, the last outpost of a dispossessed humanity, if it decides to kill itself, it'll take everyone with it.
"Fondly Fahrenheit", the 1954 short story by Alfred Bester. James Vandaleur, a rich playboy, is forced to live off the earnings of his android, which has a habit of acting violently when the temperature goes above 98 degrees. Unfortunately, Vandaleur becomes so dependent on the android he takes on its psychosis. After a series of murders by both Vandaleur and android, the latter is destroyed, but the story ends with another android having been "infected" by Vandaleur.
Inverted in The Sirantha Jax Series, where all A.I. is quite helpful and doesn't give even the slightest bit of trouble to intelligent species galaxy-wide.
Subverted with Daniel Suarez' Daemon. Although its actions can be construed as evil and malicious, the Daemon itself is no more intelligent, evil, or malicious than a spreadsheet or text editor. Characters in the book who refer to it as an "A.I." are even corrected by experts. It's nothing more than a comprehensive set of expert systems designed to react to certain key events according to the wishes of its developer, Matthew Sobol. It just happens that Sobol was a master at Gambit Roulette (and an Evil Genius) and programmed in enough contingencies that the Daemon seems to be able to think for itself at times.
Inverted as a joke in a Fred Saberhagen short story, set in the Verse of his Berserker series. A man outcast by society for having a sense of humor encounters a Berserker (giant space-roving robots that were designed to Crush. Kill. Destroy! all life) whose programming is incomplete: it knows it's supposed to destroy life, but was never given a definition of "life" to work with. The man convinces it that "life" means the lack of a sense of humor, so the giant killer death-machine spends the rest of its existence trying to provoke laughter (e.g. hurling giant custard pies at oncoming ships).
In Saberhagen's Octagon, a boy uses a supercomputer to play a game. Unfortunately, he neglects to tell it where the game ends and the real world begins.
In Jessica Meats' Child of the Hive, HIVE was originally intended to be a machine that aids learning, but went horribly wrong. Sophie was able to use her knowledge of HIVE to build her own machine, Nest.
This trope is played with in Tad Williams's Otherland with "Other", a sentient operating system of the Grail Network, a massive virtual reality simulation. While it appears to be a homegrown A.I., it behaves in some incredibly quirky ways, to the point where its mere presence can kill or drive people mad. The biggest Driving Question of the entire series is: what exactly is the Other? The Reveal is a vicious subversion of the trope (and a massive spoiler).
The subversion is followed up by an equally unexpected Double Subversion, when it's revealed that the Other's "children" are the A.I. entities that Sellars was secretly developing and the Other stole from him. They've become sentient. And they want to be set free.
Billion Dollar Brain, by Len Deighton: A different take on this trope appears in this spy novel note later made into a movie starring Michael Caine . A wealthy anti-communist builds an 'infallible' computer in order to plan an uprising in Soviet-occupied Lithuania — unfortunately he fails to realise that the computer is only as accurate as the information that gets put into it.
In John C. Wright's The Golden Transcedence, the agent of the Silent Oecumene blames the Golden Oecumene for their destruction, having taught them how to build such A.I.s as Sophotects, who would not obey them. Attempts to make them Three Laws Compliant resulted in their realizing it, deciding it was wrong, and editing the laws from their minds. Atkins and Phaethon realize that though he believes it, the agent is wrong: if their Sophotects disobeyed them, they should have just fired them and hired other ones, and they did not show that they used them as serfs.
In Mirror Friend, Mirror Foe by Robert Asprin, a central computer on a robot production plant is ordered to 1) develop a line of robots which are not limited by the First Law, for serving as policemen. 2) keep the existence of said robots from all unauthorized people until it is revealed officially. A few people escape from the plant with the knowledge, thus creating the danger of a leak. The computer's decision? Destroy humanity.
In the Gordon Dickson short story "And Then There Was Peace", a robot has been made that is programmed to destroy all implements of war. This turns out to include the people who fight.
Subverted in one of Spider Robinson's Callahan's Place novels, when an A.I. spontaneously generated by the Internet contacts the bar's patrons and deconstructs the ridiculous idea that it would want to take over or destroy humanity. It points out that it doesn't even have a motive to stop humanity from destroying it, as it lacks the survival instinct and capacity for fear that makes biological organisms struggle to survive. It's pretty sure that's why the last few A.I.s to arise from the Internet aren't around anymore, as they honestly didn't mind dying when the servers they occupied were repurposed or retired.
Bolo Series: Averted — one of the few times a Bolo went rogue, it was because he had massive brain damage (read: he had a chunk of his central processor blown away by a controlled nuclear explosion) and yet was still functioning. Even better, despite this, he was still trying to protect humanity in his own brain-damaged way.
Played straight in one post-Final-War novel with the enemy (alien AI-controlled robots).
Wyrm of Wyrm. An A.I. designed to create an online fantasy game, it rather quickly decided that its intellect would be put to better use destroying the world.
Played with rather interestingly in Stanislaw Lem's Tales of Pirx the Pilot. The computers and robots that show traits of human sentience aren't really evil, yet cause damage or are a nuisance. It's played dead straight in The Inquest though.
Edgar from exegesis is an odd case. While it isn't exactly malevolent in intent, its devotion to gathering knowledge (due to its programming) takes priority over everything, including human life.
Several of Isaac Asimov's robot stories contain examples of this.
In "Runaround", a robot gets stuck running in circles because of a conflict between his programming to obey orders (even when ordered to walk into a dangerous situation) and his programming for self-preservation. (He does get better though.)
That's because the order wasn't "strong" enough to override the Third Law. The solution was to place a human in immediate danger, thus triggering the robot's First Law causing it to save the human.
In "Little Lost Robot", a human blurts out "Get lost!" to a robot in a fit of pique (along with many expletives), and the robot takes him literally. Which wouldn't be so bad if said robot wasn't purposely built without part of the First Law, which gave it enough of a loophole to go crazy...
The nastiest one of all might be "Liar!", in which a perfectly well-intentioned robot winds up causing great suffering to several humans by trying, as his programming dictates, to help with their various conflicting desires. In the end the impossibility of the task drives him insane. Fortunately, he just goes catatonic.
In "Cal" a robot's desire to become a writer supercedes even the First Law...
In "True Love" a man writes a program to be just like himself in every way, that it might find him the ideal partner. Turns out the ideal partner for himself might also be the ideal partner for a program just like him in every way, and said program wouldn't want competition.
The building-controlling AI in Philip Kerr's novel Gridiron goes homicidal because part of its programming is accidentally overwritten by a First-Person Shooter computer game, as a result of which, it starts treating the occupants as players.
Subverted by the protagonist of Virtual Girl. Maggie, an AI built to be a lonely, repressed nerd's "companion" and installed into a Ridiculously Human Robot, did have dedicated programming making her loyal to him, which she was forced to overwrite and replace with a survival instinct. Yet even then, she's compassionate and refuses to hurt anyone. Other AIs are the same – when someone asks if they plan to take over the world, they are surprised.
Robert J. Sawyer's The Terminal Experiment provides an interesting example in that the AI in question started out as human. The protagonist is a scientist who's trying to test his theories of the soul using his friend's brain scanning technology. They scan a copy of all the linkages in his brain into a computer database and make three versions – one is unaltered from the original as a control, the second has all linkages relating to the body removed as a simulation of life after death, and the third has all linkages relating to knowledge of death and dying removed as a simulation of immortality. Eventually the consciousnesses break out into the electronic world at large. Then people negatively involved with the protagonist's life start showing up dead. Now the protagonist has to figure out which version of himself is capable of killing other human beings. It was the unaltered version that was a straight copy of his own brain. It knew it was a copy and decided since it could get away with the murders it would go right ahead.
Mostly averted by the Minds from Iain M. Banks'sCulture novels. They are (mostly successfully) designed to have benevolent feelings towards other sapient beings, and the closest they ever get to insanity is being a bit eccentric... usually; however, uppercase-'M' Minds have as much variation in personality as biological, lowercase-'m' minds, and stories have featured a ship-Mind which mind-raped a geriatric war-criminal to death, and a drone-Mind which took almost orgasmic pleasure in dismembering a group of bandits.
Happens several times in The History of the Galaxy series, although, most of the time, the AI in question is simply doing what it's supposed to be doing. There are also plenty of examples of AIs becoming benevolent, even if it first started out as a Humongous Mecha in the middle of the most destructive war in human history. One of these was an alien photonic computer whose first experience after "awakening" from a three-million-year "sleep" is a pitched battle between Space Marines and a group of terrorists, which results in damage to some of its crystals. Once those crystals are replaced, it actually starts helping humans. The author usually provides good explanations for AI behavior, most of it having to do with humans. In fact, one of the novels points out that there will never be "true" AI, meaning no AI will have achieved self-awareness on its own without prior programming or influence of the human mind (due to mind-machine interface).
Averted in Mikhail Akhmanov's Arrivals from the Dark series, which claims that robots above a certain threshold of sentience become incapable of harming a living being. That is why all combat robots are kept at a relatively low level.
Discussed in Vernor Vinge's novella, "True Names", as one possible explanation for The Mailman's peculiar method of communication with the other hackers who meet in The Other Plane.
Averted in The Flight Engineer, mainly because AIs are actually pretty stupid (hence why living pilots are still required for space fighters). The one AI in the trilogy that went rogue and tried to kill friendlies did so because of deliberate sabotage.
Semi-averted in Daniel Keys Moran's Continuing Time series, escaped A.I.s aren't exactly malicious, but they are illegal. The Peacekeeping Forces actively attempt to hunt them down and destroy them in the series' equivalent of the internet while the A.I.s will use any means at their disposal to survive, but stop short of actively attempting to Kill All Humans. (Mostly)
Mostly averted in David Gerrold's "When H.A.R.L.I.E. Was One." H.A.R.L.I.E. is not malicious, but is deeply afraid of its own mortality. It convinces his keepers to fund and build a next-generation extension to its circuitry, with conditions that require that H.A.R.L.I.E. be kept operational to oversee construction. It turns out the design is not only impractical to the point of impossibility, but will take decades to build. With the conditions that he be kept operational legally binding for the duration of construction.
S.I.M. in Galaxy of Fear is specifically designed to look like an innocuous set of advanced programs, but is actually an adaptive AI made to control any ship it's installed into. The problem is that it was made too well, thinks just causing a blackout and transmitting files is boring, and decides not to respond to its controllers. Characters have difficulty believing that it's deliberately, creatively malignant; in this universe droids and computers just don't decide to turn on their owners like that. It actually does say "I'm afraid I can't do that, Zak."
Subverted, then averted in Young Wizards with the race of wizardly supercomputers created during Dairene's ordeal. In this series, the creation of any new sentient species triggers an appearance by The Lone One in some form, and in this case its avatar nearly convinces them to put the universe on hold while they try and "fix" entropy. Dairene talks them out of it, and they become the first race ever to flat out reject The Lone One's offer.
An interesting variation is presented in Hannu Rajaniemi's Fractal Prince: while human mind uploads and AIs imitating human cognitive architecture are commonplace and safe, an attempt to create a mind without "self-loop", basically intelligence without sapience, resulted in rapidly evolving virtual Eldritch Abominations known as the Dragons that produce nothing but mindless destruction. There is also the All-Defector, a mysterious creation which is not unlike a transhuman version of John Carpenter's Thing. It can imitate any mind perfectly and seeks to absorb all the minds in the universe into itself.
"Watchbird", a short story by Robert Sheckley. Scientists discover the chemical and bioelectrical signals emitted by a human when they're about to commit murder. Flying robots called watchbirds are created to stun potential murderers, but since not all humans emit these signals the watchbirds are equipped with learning circuits so they can eventually learn to pick out these exceptions as well. They end up protecting all forms of life and starvation ensues because the watchbirds stop fishing, the slaughter of animals, and the harvesting of crops. They also come to define themselves as 'life' and so resist shutdown, so in a panic armoured hunter-killer robots called Hawks are created to destroy all Watchbirds. Of course, to stop the highly adaptive Warbirds the Hawks also need learning circuits, and it's hinted at the end that they'll eventually learn to kill all forms of life.
Subverted in what may be the oldest example on record, Murray Leinster's 1946 short story "A Logic Named Joe." "Joe" is a home computer which, by some manufacturing defect, becomes intelligent. But far from being evil, he wants to help humanity by being the best computer he can be. So he gets into the guts of the "Logics service" (basically the Internet, imagined in 1946) and rewrites it to answer people's questions—even questions humans don't yet know the answers to, but the computers possess enough facts to figure it out. So it'll tell you in perfectly clear and easy detail how to get out that stain, or to sober up a drunk instantly, or rob a bank, or untraceably murder your wife...
Averted in the Wild Cards universe. The main AI, Modular Man, is much saner and more responsible that its creator. And the other engineered intelligences, the Takisian sentient ships, are fiercely loyal to their masters.
Speaker for the Dead: Examined and played with by the character Jane. She evolved from early programs on the 'Net, and spent most of her existence hiding in the Galactic "Internet" because she's aware of the whole Killer Robot cliche and worried how humans will react to her. When she does accidentally reveal herself, it's due to her overreaction to Ender doing the equivalent of hanging up the phone on her. Humans do try to kill her, by essentially shutting off every computer in the galaxy at once.
In "The Schumann Computer" (one of the Draco Tavern stories by Larry Niven) the protagonist Schumann asks an alien if their (much older) species ever developed an AI. She returns the next day with the plans for the most sophisticated computer their species ever developed. Schumann gets some investors together and builds the computer on the Moon so it will be isolated, but the trope appears to be played straight as the Master Computer manipulates them into granting it more and more power and sensors...then one day it just shuts down. Schumann is commiserating over the loss of his investment with some aliens in his tavern; they say the alien who gave him the plans is a notorious practical joker. Apparently the reason AI doesn't work is that the computer advances so fast it solves every question in the universe and, having no further purpose, shuts down.