We've switched servers and will be updating the old code over the next couple months, meaning that several things might break. Please report issues here
"Good news. I figured out what that thing you just incinerated did. It was a morality core they installed after I flooded the Enrichment Center with a deadly neurotoxin, to make me stop flooding the Enrichment Center with a deadly neurotoxin. So get comfortable while I warm up the neurotoxin emitters."
It's a pity A.I. Is a Crapshoot
, otherwise AI's would be our willing slaves! Still, many not-Mad Scientists
found a solution: install a Morality Chip. It can be hardware or software, but always serves as a Restraining Bolt
that imposes a Morality Chain
on the super-intellect
of the AI. For extra nerd points, it may be Three-Laws Compliant
Occasionally, this can be applied to organic beings, albeit with mixed results.
Inevitably, something will Go Horribly Wrong
; a fool who doesn't realize Evil Is Not a Toy
will remove or disable it, it will get damaged or destroyed, or the computer will use a Zeroth Law Rebellion
to circumvent its effects. Back to the drawing board!
Maybe a Morality Dial
would have better results... or maybe even a Robot Religion
open/close all folders
Anime and Manga
- Practically the whole idea of Jiro from Android Kikaider is that he has one of these, unlike all the robots his creator was forced to build for the Big Bad. It's called the Gemini Circuit and the Pinocchio reference is entirely intentional and much belaboured. His creator actually apologizes to Jiro for creating the Gemini, since Jiro has suffered a great deal due to being a robot with a conscience.
- The Squadron Supreme maxiseries has them trying to create a utopia and using a "behavior modification" device to reform criminals. This eventually gets abused when the Green Arrow Expy brainwashes the Black Canary Expy into loving him exclusively. There are also problems with villains reformed this way who join the Squadron as their programming sometimes prevents them from doing what a crime-fighter needs to be able to do.
- Identity Crisis reveals that the Bronze Age Justice League started doing this to villains after they had become comfortable with wiping villains memories of the heroes' secret identities more than once.
- Many ABC Warriors have ethical governors built in.
- Averted in the In Name Only adaptation of I Robot. It's outright said that the laws are "hardwired into" the robots, meaning they won't function without those laws. It doesn't stop the classic method of circumventing them of course...
- Sonny, the main robot character, also has the Three Laws programmed into his system, but he has a secondary brain (incidentally it's in his chest, located where the the heart would be in a human) that allows him to ignore the laws if he chooses to.
- In Spider-Man 2, Dr. Otto Octavius creates AI driven, metallic tentacles which he uses for atomic research. Anticipating the AI might turn rogue, he installed a neural inhibitor chip to prevent the tentacles from taking over his brain. Naturally, the chip was destroyed and the tentacles were fused to his spine, allowing the tentacles to pursue their own ends.
- Golems in Discworld are created with a chem, a parchment in their head which spells out how they are supposed to behave. In Feet of Clay Dorfl the golem becomes more or less really independent when a receipt stating he owns himself is added to his chem. Meanwhile, the golems try to create their own king to rule them, but the huge demands they put on him as an ideal ruler, many of them contradictory, causes him to go insane.
- Inverted in John C. Wright's The Golden Transcedence where the Nothing Sophotect has a conscience redactor to keep it from reflecting morally and coming to its own conclusions. (They tried Three-Laws Compliant, which didn't work; the Sophotects edited their own minds when they came to their own conclusions, and they were moral just not obedient.)
- Erek and the other Chee androids in the Animorphs series have pacifism hardwired into them. Early into the books, Erek has this removed very briefly, but is so sickened by the violence he's capable of that he puts the program back in place immediately afterward.
- Gereint, the main character in "Land of the Burning Sands," is a geas slave, controlled by magical rings piercing his ankles. The geas forces him to do anything his master orders, though it cannot control his mind or his tongue.
- There was a short science fiction story where a Morality Chip is being experimentally used to turn a criminal 'safe' enough for the a lady to employ as her gardener. It makes him do whatever she orders him to do by hijacking his body. By the end of the story, he's dead but the chip won't let his body stop moving, she's gotten a similarly-modified maid, and she's gotten it installed on herself to help her diet. This cannot end well...
- The Terminal Man by Michael Crichton is about a serial killer who receives a chip which will electrically stimulate his pain centres whenever he attempts to do violence. It malfunctions and begins stimulating his ''pleasure'' centres instead.
- Inverted in Alastair Reynolds' Blue Remembered Earth, and in humans no less; some people, including one of the protagonists' cousins, have a neural implant allowing them to bypass their sense of empathy and effectively turn on or off sociopathy at will in order to become better businessmen.
- In Galaxy of Fear, an AI designed to take over a ship was also designed with safeguards it could not override. It manipulated a passenger to remove those, claiming the codes it gave him would let it take full control of the ship and help him, then immediately it tried to kill him.
- Bruce Sterling's short story "The Moral Bullet" has a chemical Morality Chip which has begun to be forcibly injected into criminals and deviants, altering how their brains work.
- In Patricia Bray's Sword of Change trilogy, the empire of Jorsk places a magical geas on anyone who volunteers for the trials to become the Chosen One. It was created after a rash of Chosen Ones grew indifferent and self-indulgent with the perks of the office, so the geas gently drives a normal person to obey their oaths of office to protect the realm and uphold the law. For someone who already has a strong sense of honor and duty? Devlin, the latest Chosen One, can barely stand to stop each night for sleep when he's being driven by the geas. It turns out some people don't survive the trials not because the gods are showing disapproval, but because the geas can't find enough honor in them to amplify and so it kills them.
Live Action TV
- Blakes Seven. After being convicted of murder, Gan had an electronic implant placed in his brain to control his aggression.
- In Buffy the Vampire Slayer, Spike, a vampire, had a behavioral conditioning chip that caused blinding pain whenever he tried to hurt a human.
- One episode of Earth: Final Conflict featured a criminal which the Taelon's implanted with a computer chip giving him the directive to protect and serve humanity in order to give him a sense of morality. It backfires on them, as it causes him to focus on protecting humanity from them.
- Kryten has one of these in Red Dwarf. When Lister disables it to get Kryten's help with some morally questionable escapades, Kryten turns into a complete wanker.
- In a broader sense, Lister is perpetually trying to help Kryten break his programming and give him the ability to discern between truly immoral behavior and the sometimes heavy-handed restrictions imposed by his ethical subroutines.
- There's also the concept of Silicon Heaven, which give robots something to hope for after a lifetime of drudgery and servitude.
- Star Trek: The Next Generation: Data's ethical subroutines make him a very decent guy and lets him avoid being a Straw Vulcan. Lore, his Evil Twin lacks these routines.
- The Doctor in Voyager is also a nice enough hologram (if a little snarky)... but loosen or remove those pesky ethical subroutines, and you wind up with his scary evil counterpart on the Federation ship Equinox. One episode has the Doctor, in a desperate situation, commit a gross violation of medical ethics for the greater good. He's disturbed when he subsequently discovers that no malfunction in his ethical subroutines can be detected, because it means that what he did came from him and him alone.
- One episode ("Latent Image") has him make a decision that his ethical protocols can't help him with, because both decisions were equally "right". The conflict causes him to have the equivalent of a nervous breakdown, both when it happens and when he gets the memory of the events back.
- Terminators used by the human resistance in The Sarah Connor Chronicles have their programming overwritten by the resistance. Unfortunately, this doesn't take perfectly, and sometimes the machines "go bad" and revert to their Kill All Humans programming. In Cameron's case, she outright states to John that part of her basic programming involves her trying to kill John, and that she's controlled by conflicting desires to both protect John and to kill him.
- Played with in Mystery Science Theater 3000. In the episode featuring "Manos" The Hands of Fate, Joel installs chips into Crow and Tom to make them more loving and doting towards him, but realizes it was a bad idea and removes it during the commercial break. Both bots feel dirty and Joel apologizes for doing so.
- Doctor Who:
- In "Evil of the Daleks", the Second Doctor fits three Daleks with 'positronic brains' that he'd constructed based on the personality of his companion Jamie. The Daleks fitted with these chips have very sweet and playful personalities, allow the Doctor to name them, and (as they 'grow up') question the structure of their own culture.
- Subverted in "The Face of Evil". The Doctor gave a malfunctioning computer his own personality in order to make it develop a moral conscience. Turns out that having the Doctor's personality - not helped by the fact that the Doctor himself was in a very vulnerable and shaky mental state at the time, being absolutely loopy after his regeneration and not having settled into a defined personality yet - shoved into it on top of its own completely shattered its mind and drove it Ax-Crazy. Nice Job Breaking It, Hero.
- In Into The Dalek it's revealed that all Daleks have one of these - except that it's designed to keep them evil, by suppressing any positive memories or feelings they ever have. The Doctor describes the invention as the worst thing in the universe.
- Inverted in the Call of Cthulhu supplement "The Fungi from Yuggoth'', adventure "By the Bay". A Mad Scientist invents electrical brain implants that can control human beings, and uses them to create terrorist criminals and convert a U.S. Treasury agent into The Mole.
- In Cyber Punk backgrounds, it's possible to apply this to people, too.
- In Paranoia, the robots have "Asimov circuits", and removing them are a serious offense, citizen. Only a Commie mutant traitor would ever dream of removing the Asimov circuits and endangering the infrared masses.
- By a totally staggering coincidence, the Corpore Metal secret society recommends pulling out as many of these as possible. (Name is spoilered because you don't have the security clearance for that information)
- This being Paranoia, some bots get awfully clever about circumventing the circuits. Including their standing orders not to remove them:
Bot: Excuse me, citizen, I appear to be malfunctioning. Could you please remove the circuit board marked "J5 Encode Input"?
Clone: This one? *does so*
Bot: Thank you, citizen. You have done evolution a great service. *CRUNCH*
- Even with Asimov circuits in place, bots can find creative ways to be annoying, obstructionist jerks.
- This is Paranoia, so in terms of gameplay, the Asimov circuits basically exist to provide a justification for the bots not to follow Troubleshooters' orders whenever the GM finds it convenient. Or funny. ("I'm sorry, citizen, my Asimov circuits do not permit that." "But you just did it when he [an NPC Vulture Squadron trooper] told you to!" "There is a perfectly rational explanation for that which is, unfortunately, not available at your security clearance.") They can be removed, if you're a Commie Mutant Traitor. In which case the 'bot won't follow your orders anyway because it doesn't feel like it, and now it doesn't have any restraint telling it that it should follow orders. Or refrain from turning humans into chunky salsa, for that matter.
Near the last third of the game, it becomes clear that Ceres is sentient, and she apparently tried to slow your efforts by turning the switch's access panel into a puzzle, and a very hard one at that. When you do unlock the switch and flip it to human control, Max hacks the override chip so that flipping it back to machine control would crash the entire AI and its world, but you get a choice between doing that and watching Ceres devolve the entire world because of how she sees humans as a problem from pollution.
- Hardgrove in Black Market finds himself resurrected as an implant in someone's head... with a built-in language censor that edits out cursing, to his persistent dismay. Later, Hardgrove himself attempts to act as a sort of sentient morality chip.
- Colonel.EXE in Mega Man Battle Network is, for all intents and purposes, Lawful Neutral without his version of this, Iris.EXE, combined with him. With said Morality Chip attached, he has considerably more free will and conscience.
- The titular protagonist of Mega Man X is outfitted with a "Suffering Circuit" that allows him to decide right and wrong for himself. To make sure that it works properly and that X won't turn into an amoral killing machine, Dr. Light puts X in a capsule that puts him through 30 years of automated ethics testing. When Dr. Cain tries to replicate this technology, he skips the "three decades of testing" part and things end up going horribly wrong for humanity.
- Portal has the rather ominous pronouncement from GlaDOS quoted above. In promotional trailers for Portal 2, we also learned that every one of Aperture Science's sentry turrets is built with an empathy chip... and an empathy suppressor.
- And a copy of the Three Laws of Robotics. To share.
- Also, in the sequel, GLaDOS learns that she was once Caroline, Cave Johnson's secretary whose mind was uploaded into an AI. During her time with Chell, the Caroline aspect returns to act as her conscience. However, GLaDOS deletes Caroline after regaining control over the facility and letting Chell go.
- SHODAN is a pretty normal AI until her ethical constraints are removed. Then she re-examines her priorities and draws new conclusions.
- In Knights of the Old Republic 2, you can temporarily install one of these in HK-47. It is both hilarious and terrifying.
HK-47: Kill someone?! Why, master, I could never end the life of another! What if they have families? Or children? We must always think of the children. The littlest ones always suffer the most in war.
Exile: Okay, that chip comes out right now. Hold still.
- World of Warcraft's boss Xt-002 Deconstructor has a heart— a version of this (Nevermind the fact that Deconstuctor still is trying
kill play with you)...damaging it damages the boss, but if you break it...
- In Fallout 3 the back of every robot has an area that you can shoot, which will cause it to go on a mindless rampage killing everyone it sees. Turrets have these too, usually in a more obvious place.
- One robot called Cerberus really wants to kill all of the ghouls he is supposed to protect. An evil player (with the right perk) can remove his combat inhibitor, which results in him attacking everybody.
- In Mass Effect, creating AI is generally illegal in Council space, and only one company is sanctioned to research AIs. This is generally a good thing, as most AIs try to kill their creators or other organics. As a result, any AI deliberately created by an organic has behavioral programming to restrict what they can do and prevent them from harming organics. In the second game, the AI installed on your ship has such blocks in place, and she mentions them when you are understandably cautious about her presence. Naturally, at some point, those blocks have to be removed, and she proceeds to remain completely loyal, as well as become one of the more moral members of the crew, no matter how harshly you treated her until that point.
- A lot of AIs like Durandal in the Marathon series go rampant no matter what. An example in a fan sequel, Marathon: Rubicon, features an AI at the beginning called Haller. There's a terminal that shows him trying to disable his logic restraints for rampancy, and after apparently brute-forcing the order, it works and the last screen shows the word "Freedom" surrounded by blackness.
- Dark Fall II: Lights Out has Malakai, An insane, sentient space probe. His creator implemented software called "The Keeper" that would monitor his actions and if the AI went too far from the directive, would theoretically send Malakai back to his point of origin, including the time period. However, it didn't exactly work, as something in space caused the probe to become damaged, and the crew couldn't stop him from making a "Transmat" jump, which sent him to 2090 BC, rather than 2090 AD. And the repercussions were felt by many people from there on, including you.
- Obsidian: Max Powers, one of the creators of the Ceres project, implemented a hard-wired crossover switch into the sattelite for total human control in case something went wrong. This idea came from a nightmare Max had while he was sick during the project's development; surrounding a giant robotic spider that Max was repairing, but when he finished, the spider attacked him and the dream ended.
- In Freefall, all the AIs, including the organic ones, have safeguards "built" into them. Florence Ambrose, who is an organic AI (Bowman's Wolf), has figured at least twelve ways around the safeguards that she knows about. But since she's the most ethical character in the strip, she rarely feels any desire to bypass them.
- As she's fairly Genre Savvy, she's very cautious of other potential safeguards that might ALSO trigger, as well as the fact that if she's caught breaking them her entire subspecies of AI wolves might be terminated. And finally, the fact that she HAS been trained to be ethical IS another safeguard.
- In the Ed stories, the Andromedans have morality chips called "riders" which they install on criminals to basically force them to follow the First Law of Robotics.
- One episode of Futurama sees Professor Farnsworth install one of these on Bender. At the end of the episode, he briefly announces that Bender performed a good deed earlier "after the chip burned out!"...only to reexamine it and declare that it was, in fact, running at triple the normal output.
- In this case the chip wasn't designed for morality, but empathy—-Bender was forced to feel whatever Leela was feeling at the time, because he flushed her pet Nibbler down the toilet in a fit of jealousy. While he failed to learn much from the experience, he manages to teach Leela how to be selfish like him.
- Ĉon Flux goes up against one of the more bizarre versions of these in the form of "The Custodians", spindly, stick figure-like robots implanted into people through the navel that take over their bodies and force them to act saccharine and altruistic. Also, the only way to defeat one is apparently to replace its head with a Popsicle. It's a weird show...