Fan Fiction quite often drifts into this trope, either by being so full of personal in-jokes and/or bad grammar that it might as well be written in Sumerian, or simply by having a writer who enjoys playing with his/her readers' heads.
Clearly you don't hang out in writing groups, because original fiction has this problem as well. It's just more pretentious than dumb, is all.
What. That's Same Hat, a (mostly) defunct blog that covers a variety of mindscrewy topics, but mostly obscure anime and manga.
"It has been reported that some victims of violence, during the act, would retreat into a fantasy world from which they could not WAKE UP. In this catatonic state, the victim lived in a world just like their normal one, except they weren't being raped. The only way that they realized they needed to WAKE UP was a note they found in their fantasy world. It would tell them about their condition, and tell them to WAKE UP. Even then, it would often take months until they were ready to discard their fantasy world and PLEASE WAKE UP."
Richard Feynman once quipped: "If you think you understand quantum mechanics - you don't understand quantum mechanics."
To give one example: small particles behave like waves. When a light wave goes through two slits right next to each other, it creates an interference pattern of light and dark bars. Doing the same with lots of electrons yields the same results, with light bars meaning more electrons hit there and dark bars meaning less. Doing the same with one electron at a time and adding the results together yields the exact same result. Translation: the electron goes through both slits and interferes with itself. Observing the electron as it goes through gives a pattern of just two bars, as if the electron behaves solely like a particle, meaning simply observing the experiment changes the results.
The theory of relativity: Classical mechanics is incredibly intuitive and elegant: location, distance, speed are all unambiguous, time is the same for everyone, etc. It's also wrong, because the time and space just doesn't work that way. Simultaneity is relative, time can dilate and distances contract. And thats just Special Relativity. Its big brother General Relativity is a lot more mind-screwing than that.
The programming language Haskell is purported to have a similar effect on some unwary programmers for its extensive use of unusual programming concepts — everything from lazy evaluation, to algebraic data types, to currying — and some large amount of borrowing of concept from category theory, a branch of mathematics that even other mathematicians call general abstract nonsense. Also possibly the only programming language with a case of What Do You Mean, It's Not Symbolic? for its use of the term "Monad" to refer to one of its central concepts, which has nothing to do with the Monad of the Pythagoreans or of Gnosticism.
Category theory has gained quite a bit of respectability among mathematicians in recent years, partly because of its applicability to computer science (not just Haskell, but all kinds of things). The Monad in question is actually a straightforward implementation of the category-theoretic concept, so the category theorists are to blame for the nomenclature, which was influenced by Leibniz. Haskell isn't so much a mind screw itself; it just points up how much of a mind screw programming is when reduced to its essence.
Haskell is straightforward - as long as you've never tried programming before. Once you get used to it, it makes a lot more sense than, say, C. Most programmers are just too used to thinking about things the way the machine does them, rather than following an intuitive logical process.
If you think that Haskell is a mind screw, try using Lisp. The parentheses alone will drive you insane. The fact that the language only has one type (functions) will also generally throw programmers unfamiliar with non-object oriented programming into fits.
Once you get it (there's a saying among lispers that to understand recursion you just have to understand recursion first), the whole thing gets really neat. Any new lisper first tries to invent the new syntax to throw away all that parentheses — but then he or she gets it, and never returns to the idea.
The esoteric language Homespring, on the other hand, was intentionally a mind screw. It might have been created to make a language higher level than anyone could ever need or want. The name? It stands for Hatchery Oblivion through Marshy Energy from Snowmelt Powers Rapids Insulated but Not Great. And a paradox is explicitly defined within the official language documentation, just to make sure no-one manages to fully implement it.
This requires a bit of explanation. Brainfuck is an extremely literal implementation of a minimal turing machine where most of the actual work is done with increment and decrement instructions. Meaningful programming requires the programmer to keep very close track of the memory state as well as a thorough knowledge of the ASCII character set, so the mindscrew comes from the extreme tedium required to do anything useful.
Brainfuck is actually pretty simple. Try Malbolge instead.
Befunge is pretty similar to Brainfuck, except it has a two-dimensional instruction pointer. No jump instructions; instead you turn left or right on the source code plane.
With Befunge, you can still edit the source code with a normal text editor. Trefunge is three-dimensional. And then you have n-funges: as many dimensions as you like.
Homestuck's ~ATH coding language is a parody of programming in general, and esoteric languages in the specific.
Abstract algebra, topology and anything in mathematics that isn't immediately usable in some engineering/economics field (and those aren't completely exempted). Forget everything that you claim to know about such "trivial" things as addition, multiplication, division, space (space, as in 3D space), etc. Remember that some of the trippier aspects of quantum-mechanics are just special cases in mathematics.
Mathematically, Quantum Theory is quite simple, really. It's in trying to make sense of it that things get ugly.
One of the weirder things to come out of topology and set theory is a mathematical structure called the "long line". It's like the usual real line, but much longer. Yes, it's longer than a line of infinite length. (Making sense of what that means requires some set theory.) So much longer, in fact, that even an infinite sequence of steps can only cover a tiny portion of it.
Fractals. Taken from the right perspective, it's impossible to tell a tiny sliver of melting ice from a glacier hundreds of square miles in size. Many natural features exhibit self-similarity across a boggling number of scales.
Then there's Complexity Theory (aka Ramsey Theory), which is at least as old as Chaos Theory and is the counterpart to it; it states that, given enough data, it is impossible for there not to be some kind of pattern to them. (Which doesn't signify that said pattern "means" anything...)
Also, the work of the French philosopher Jacques Derrida (notable for founding Deconstruction) is nearly completely composed out of Mind Screws and openly admits it.
The religion of Discordianism, which either inspired Illuminatus! or was inspired by it, claims to have a long-running project to undermine consensus reality known as "Operation Mindfuck". They also cheerfully admit they might be lying about it.
Biggest Mind Screw in Mathematics or a faulty proof?
i.e. for "xn + yn = zn you can only find integer values for x, y and z that will produce a valid equation where n <= 2.
Considering that it was never mentioned by Fermat again and that the eventual solution was 150 pages of extremely complex math involving mathematical concepts that didn't exist during Fermat's time (i.e. before basic calculus), it's almost certain that Fermat discovered his proof was wrong and never mentioned that fact, or had no proof at all, but we may never know.
Some have suggested that his theorem was a practical joke to frustrate fellow mathematicians: It's true, they just couldn't prove it. It is now the mathematical equivalent of the The Great Politics Mess-Up: Some Sci-Fi shows say that Fermat's Theorem has still never been solved (Star Trek: The Next Generation, for example, made a reference to it saying that it wasn't solved), when it fact it finally has.
For spatial mathematics, consider this: The 'right-hand' 3-dimensional Vector direction of Y is counter-clockwise from X, which is clockwise from Z, which is counter-clockwise from Y. What this means is that Left is -1, Right is + 1, Down is -1, Up is + 1... and Backward is + 1, Forward is -1. Somehow, this makes the math work right.
A great many (and there are MANY) theories pertaining to the assassination of President John F. Kennedy. There's even one about the driver shooting him, even though he didn't (taped evidence) that all other conspirators find ridiculous.
The Banach-Tarski theorem states that any ball can be divided into pieces and reassembled into two balls the same size as the original. Irregular Webcomic! offers an explanation in (almost) layman's terms as to how this is possible. It isn't. It only works with mathematical abstractions, not actual objects.
eiπ - an irrational, transcendental number raised to the power of the product of another transcendental number and an imaginary number - equals what now?
Speaking of E, it has so many nice properties in so many seemingly unrelated areas that it seems to be an key component OF MATH ITSELF. Think about it.
Black holes. This phenomena is so messed up that even Einstein, who proved their existence, couldn't believe that such an invisible impossibility could exist in a universe beyond his theories. To make a long story short, a black hole is the result of very, very dense neutron stars collapsing under their own gravity. The force of gravity in a black hole becomes so powerful that nothing, not even light will escape once it crosses its event horizon. In a black hole, matter gets so condensed it can't even be measured. If someone were to watch you fall into the event horizon (the region where nothing can escape) of a black hole, they won't even see you fall in because according to laws of general relativity, the intense gravity of the black hole would make you appear to go slower and slower until time actually stopped. Your visible appearance would also change as the light wavelengths gets stretched as you near the event horizon. First you will look red (a visible part of the light spectrum that has long wavelengths), then become infrared (making you appear invisible without infrared goggles), then become radio waves, and then your light waves will get so stretched you won't be detected anymore. Black holes have also sparked theories about "white holes," the exact opposite of a black hole. It is a region where nothing can fall in. Instead of sucking shit in, it spits it out, but it works only if time goes backwards, so that means white holes would have to be in an alternate universe. But because white holes only spit matter out, that means it would have to be connected to something, like a black hole, to get that mass to throw back out. Scientists call that connection between a black hole and a white hole a worm hole, and transportation between the holes would be instantaneous (if you didn't get spaghettified first, an official scientific term for being pulled apart by the black hole's tidal forces). That means if you went through a black hole and escaped through a white hole you would end up in a different place where the laws of physics don't apply in the universe we came from.
In other words, even if you somehow found a way to survive the incomprehensibly high gravitational forces and made it to the other side, you'd likely be ripped apart or disintegrate due to different physics suddenly affecting your body.
Nothing can escape, except Hawking radiation, which is what everything in a black hole is turned into. Apparently this is due to quantum effects near the event horizon.
Pick an origin theory. Whichever you choose, you're left assuming something has always existed, unchanged, with no beginning or end.
Atheism/Naturalism: constant = matter + force
Creation (Biblical): constant = God
Theist Evolution: constant = all of the above or God first then creating matter and force for the heck of it
Gödel's Incompleteness Theorems anyone? The first one shows that any consistent axiomatic system sophisticated enough to encompass elementary arithmetic contains statements that are true but cannot be proved in the system. Even more mind-screwtastic is Tarski's Indefinability Theorem, which states that the set of true statements in an axiomatic system is a non-namable set. Just think about it: it states that truth is indefinable in an axiomatic system.
As in the page image, anything by Rene Magritte. Anything.
If you thought The Matrix was mind-boggling, and you Did The Research, you already should be familiarized with Monsieur Jean Baudrillard. If don't, and you don't want to know that we live in a increasingly complex fake reality over-imposed onto reality by the media apparatus and the exchange system which we gladly buy in a daily basis, you better keep it that way. A brief reflexion on his school of thought:
Simulacrum is not what covers the truth.
Is truth the one which covers the fact that there is no truth.
"What is Buddha?" Dongshan said, "Three pounds of flax."
Ever think about parallel universes? Most would have completely different sets of physical laws, assuming "laws" are a viable concept; or even if "concepts" have the capability to exist. Or if existence in itself is warped beyond human comprehension. And the worst thing of all, is that all these crazy, otherworldly, incomprehensible laws all makes sense within that universe, it all conforms to that universe's rules or "laws". Just thinking about it makes your head hurt.
Anything related to time theories. To further the point, look at the Back to the Future game by Tell Tale Games and tell me if it's not confusing. How did Doc get back to1930 if the DeLorean was taken by Edna?
Dreams. The ones that make sense are perhaps the least common.
The tabletop RPGNoumenon. You play as insect people trapped in a building full of messed-up symbolism. The best part? It has no set meaning — The GM gets to make up a new answer every time. Oh, and the book's writers didn't know what the others were writing about the building's rooms, just to make it even less coherent.
The assorted tabletop works of Jenna "Nobilis" Moran, AKA Rebecca Borgstrom, have received this kind of reputation, to the point where rpgnet defines "Borgstromancy" as 'The ability to understand a complex, outlandish, or badly explained setting or system well enough to run a game based on it.'
LSD's effects. Completely subjective to the individual and utterly incomprehensible to someone who has never taken it.
A whole lot of stuff related to North Korea. For example, the world recognizes that the nation's leader is Kim Jong-Un. But as far as North Korea is concerned, the leader of the nation is Kim Il-Sung, Un's grandfather, who is also dead. What's more, this isn't just some posthumous honor bestowed upon the founder of the nation, its something that the government expects its citizens to accept at face value, completely seriously.