In general, the details of a story's setting basically serve two purposes: to grant the audience Willing Suspension of Disbelief, and to enable the plot to happen. This can range from technobabble to incredibly intricate rules of science or magic or simply society, but on any level the rules form the setting in which the story takes place.
In some works, such rules of the setting not only shape the story, they are the story. The characters may or may not know what they are, and intend to explore how they interact or outright abuse them for their own advantage. It becomes a form of mystery, with a Driving Question of How does this work and how can it be used? (rather than Who dunn it?, Who am I?, Where am I?, or What's in the box?), and the longer the story, the more answers to both facets of this question are likely to be found.
By their nature, such stories tend to encourage certain strains of the Science Hero and Guile Hero, and the Rules Lawyer is more than welcome. The audience of such stories tends to revel in Complexity Addiction and may find themselves having to take notes as necessary, because even Mr. Exposition will probably expect them to keep up.
Refer to Minovsky Physics, Magic A Is Magic A, Rule Magic, and Three Laws-Compliant - these types of stories are much more frequently Speculative Fiction than other stories with puzzle elements, likely because much of the reader's engagement in the story comes from trying to anticipate and deduce new properties of the mystery and new ways to use it to solve problems, and this is difficult for the writer to maintain if the real world contains spoilers. (Those that do have a real-world setting tend to involve specialist fields of knowledge, such as law, psychology, or mathematics, and put special effort into Showing Their Work.)
Compare the Lord British Postulate and Sequence Breaking, which result from the audience engaging themselves in game mechanics in the same way. Contrast New Rules as the Plot Demands. May involve Loophole Abuse. Runs entirely on the Cool of Rule.
Main Plot Examples:
Anime and Manga
- Death Note - Light knows the rules (and learns new ones whenever Ryuk feels like telling him), and L (and later Near and Mello) intend to work them out in order to stop him.
- Future Diary - it's basically a big battle crossed with Scry vs. Scry, with each of the eponymous future diaries working under shared rules with individual special restrictions.
- Lost+Brain has been compared to 'Death Note but with hypnosis'.
- This is the Liar Game. It's a game where you lie.
- Kaiji features rules that have been laid out specifically for competition purposes, in which law-abusing is expected and even encouraged but law-breaking will result in costly disqualification.
- Primer. In this case the rules are complicated enough that the audience may not understand the explanation entirely... but it's okay, the protagonists are engineers.
- Exam. Sort of. We're told the only "rules" right at the start, but the main point of the film is working out what those rules meant.
- Isaac Asimov established the Three Laws of Robotics principally to write stories of this type. The page-quote synopsis of the stories of I, Robot is pretty fair.
- From The Unorthodox Engineers, there was a short SF story called "The Pen and the Dark", by Colin Kapp, in which Sufficiently Advanced Alien Lost Technology is causing bizarre problems that defy the laws of physics and it's the job of the protagonist to work out how to solve them using only human science.
- Kenny vs. Spenny: Each episode has the two friends competing at something, each trying to bend the rules in their favor. Of course, that's when they don't just cheat.
- From a meta point-of-view, Magic: The Gathering has basic rules and the idea is to bend them with the effects of various cards.
- The Other Wiki has a list of games with mutable rules
"In some games, making or altering the rules is part of the game. In many (but not all) of such games, manipulating the rules to one's advantage is the best strategy to win. Persuasion becomes a key technique, and rules lawyering is often encouraged."
- In Fluxx and its derivatives (EcoFluxx, ZombieFluxx, &c), the cards you play may set new rules or victory conditions when played that last until the card is removed.
- Casual Puzzle Platformers tend to do this intentionally, giving the player a rule-guided environment to survive through, goals within it to obtain, and limited tools to use creatively to do it (run, jump, and gimmick).
- Tower Of Heaven is particularly notable. You want to get to the god at the top of the tower, and he keeps giving you more rules to obey as you ascend - "Thou shalt not touch living things", "Thou shalt not touch blocks or walls from the side", "Thou shalt not walk left"... "Thou shalt not review the Book of Laws."
- Depict1 deliberately lies about the rules.
- Erfworld takes a real world wargamer and summons him into a world that literally runs that way.
Anime and Manga
- Code Geass - the various geasses, and their limits and capabilities.
- Yoshihiro Togashi is fond of this trope.
- In Yu Yu Hakusho, a small group of humans gain the ability to create "territories," where they can create physical rules to the space and people within it. Often (but not always), the creator of the territory can only be beaten by getting him- or herself to break one's own rules. The most notable case is Kaito, whose rules are, "No violence is permitted" and "Every 30 seconds, a letter is banned from use in a spoken word." Kurama defeats him by waiting until nothing can be spoken, then tickling him to get him to laugh.
- In Hunter × Hunter, there are a lot of characters with complicated rules for their powers—characters can decide what sort of powers they'll have, and the more conditions there are, the more capable the power will be. All of the best fighters in this series are Genre Savvy enough to reveal as little as possible (some choose to not use their powers at all until necessary), so victory in a fight largely comes down to who can figure out the opponent's powers first. So far, there has only been one villain in the series whose power contains fewer than 6 rules.
- Mistborn - Allomancy, Ferruchemy, and Hemalurgy and the inherent abilities of kandra and koloss combine in some surprising ways. The Lord Ruler managed to make himself practically immortal this way. (Brandon Sanderson admits to having been inspired by Magic: The Gathering.)