Older than Television: E. E. “Doc” Smith's Arisians (in the Lensman novels) do exactly this to interlopers who enter their space uninvited or in violation of previous warnings. All the bad, wrong or evil things they have ever done are dredged up to haunt them, and they can adjust the intensity. Helmuth gets off light with only a brief dose, but he's still shaken from the experience and heeds their warning not to come back. On the full dose, people go insane to the point that they kill themselves. And for those races too cold-minded to be Mind Raped, they can skip the pleasantries and turn them into People Puppets.
In the story "The Blood Red Game", the surviving remnants of a dying universe engage in a type of mental competition with an alien race who does this as their primary way of resolving conflict. Since the aliens' physical weapons would obliterate the refugees, the refugees agree to play this game. Basically, whoever does the most Mind Rape on their opponents wins.
A mild version of this is a favorite tactic of the elves in Terry Pratchett's Discworld novel Lords and Ladies; they use their "glamour" (which is essentially a form of psychic domination) to make all but the most strong-willed mortals (or anyone wearing iron) feel worthless and powerless.
In Eric, the new Demon King Astfgl has worked it out that Hell's traditional punishments - burning, etc. - are useless for tormenting the damned, who have no bodies. He substitutes relentless mind-numbing boredom, like having a demon show you an interminable slideshow of his vacation to the Fifth Circle.
What's interesting is, the King of Hell was actually ripping off Humans themselves! In other Discworld novels, it's pointed out that humans invented concepts such as mediocrity, dullness, etc. (Compare the Real Is Brown trope to, say, any photo of nature, or just go to a park or something). For example, forcing the Sisyphus expy to listen to hours of boulder pushing safety regulations rather than being allowed to just get on with pushing the boulder. Hell, even the torturers felt tortured.
The demon Crowley is a master of this technique in Good Omens. He has realised that being stuck in a traffic jam on the M25, or subjected to a mobile phone outage, or suckered into accepting a phone call from a double glazing salesperson, makes life just that little bit more of a Hell on Earth.
A less funny version of this trope is Played for Drama in Night Watch; many of the victims of the Unmentionables have been tortured to the point of permanent insanity, leading Vimes to perform a Mercy Kill for the worst cases when he tries to bust them all out.
While, in the books, Legilimency is used only as a Mind Probe, the fifth Harry Potter film suggests that Voldemort uses it to inflict mental torture as an end in itself.
Coming near a Dementor will cause a Mind Rape-like effect to occur; they are used as guards in the Wizard prison of Azkaban to sap the prisoners' will to escape. And that's just when they are standing near you. If they actually attack, they can suck your soulright from your mouth.
Boggarts do something similar, picking your greatest fear out of your mind and then assuming that form.
This seems to be what happened to Ginny in the second book in regards to Riddle's diary. It was one of Voldemort's Horcruxes. Get a little too attached to one and it'll start playing hobnob with your head. That's what happened to Ron after wearing Slytherin's locket (another Horcrux). It started picking at his fears, to the point that Ron broke off from Harry and Hermione for a while, only to come to his senses at the last moment and pull off the save.
It also happened to Dumbledore after he drank the potion in the cave. Although the potion is supposed to burn your throat something terrible, it mainly seems to attack the psyche with your worst recollections.
Jane and her brother Alec each have their own special Mind Rape powers. Jane's is pain while Alec cuts off all a person's senses.
Jasper. He doesn't use his power this way, but if he did ...
Many of the vampires with mental abilities can fit this trope. The Volturi also have Chelsea, who could strengthen or weaken your emotional bonds to others, taking away your free will. Renesmee can insert her thoughts into your mind. And even Edward's and Aro's ability to read thoughts can feel rapey since it invades the privacy of your mind, preventing you from keeping any secrets from them.
A similar case is the basilisk in Peter David's Knight Life. She reveals that a basilisk's gaze doesn't kill you in and of itself: it lets you see yourself for who you are, everything about yourself, even the things that are hidden from you. Most victims, faced with everything they didn't want to know about themselves, willingly submit to being eaten.
Northern Lights: The way Lyra describes how it feels when an attacker touches her daemon, she could very well be describing a rape:
"It was as if an alien hand had reached right inside where no hand had a right to be, and wrenched at something deep and precious. She felt faint, sick, dizzy, disgusted, limp with shock. ... It wasn't allowed. Not supposed to touch. Wrong..."
To further exemplify this comparison, her lover Will does the same thing on purpose. Only this time, she enjoys it. Context is everything.
The Mule in Asimov'sFoundation series has this ability, and ruthlessly uses it on the poor Second Foundation decoy who ends up completely brain-dead by the end of it. In one scene, he muses on how he could use his mind-controlling abilities for physical conquest, which wouldn't count as rape since the subject would genuinely feel nothing but complete love and devotion to him — but doesn't, because... he didn't choose his nickname due to his stubbornness or physical strength...
The Sword of Shannara from the eponymous novel, shows the person the absolute truth, stripped of any sort of perspective—every little lie one has ever told oneself or another is stripped away. It is the only weapon that can harm the Big Bad as he is keeping himself alive through sheer effort of will and self delusion.
Incarnations of Immortality: In On a Pale Horse, Luna Kaftan, the main female character, confesses, "I have fornicated with a demon of Hell." She doesn't reveal until much later in the novel that the demon violated her mind and soul, but not her body.
Mucking about for any reason in someone's head (no matter the intent) in The Dresden Files will usually cause permanent mental damage. There's a reason people who do it are usually summarily executed.
Also, White Court vampires usually do perform both the Mind and regular varieties, though their preference varies by family. Vampires from House Skavis cause people to feel despair until they commit suicide, those from House Malvora cause fear until people die of a heart attack, and the Raiths cause lust, usually seducing and feeding off people's souls during sex. Lara Raith both mind rapes and regular rapes her own father so hard that his mind is completely destroyed. To be fair, he had done the same to her and he really had it coming.
In Grave Peril Murphy ends up being mind raped from a nightmare demon. It was so bad that in the next book Murphy was suffering from paranoia, insomnia and substance abuse and Harry outright states that she was raped. It took a while for her to overcome it and even in Aftermath she still has a near paralyzing fear of mind magic.
In the same book, Harry witnesses victims of some form of Mind Rape involving wrapping the mind and soul in something that manifests itself to his Sight as similar to barbed wire. The victims are catatonic and in constant pain.
Also, in Small Favor, Mab mindrapes Harry to keep him from using fire magic, which would draw Summer's assassins down on him immediately.
In the 11th book, Dresden encounters something sufficiently nasty that merely looking at it with his Third Eye instantly mindrapes him BAD.
Not to mention the traitor on the White Council having about three-quarters of the younger Wardens of the Council in the grip of mind control, and subtly influencing the thinking of the entire Senior Council through enchantments and potions for years. Said traitor even turns this into actual rape, to a degree. Luccio was only sleeping with Harry because said traitor's mind control was making her attracted to Harry in the first place.
And let's not forget Thomas getting taken apart physically, mentally and emotionally by the Big Bad. Mind rape seems to be a theme in Turn Coat.
The White Watch in Jesse Hajicek's The God Eaters conduct mental "Surveys," in which a member of the Watch is searching a person's mind for magical ability or for information. Often, the Surveys are described as painful Mind Rapes, and in some cases used as instruments of torture. The galling part is that later, you find out that it's entirely possible to do it painlessly, they just don't care or weren't trained to do so.
In the Conan the Barbarian story "The People of the Black Circle" (1934), a princess is forced to relive all her past lives — many of which, it is implied, suffered actual rape among other degradations.
The necromancer Vargûl Ashnazai in Nightrunner has the ability to force visions on people. The hero Alec is held captive and treated nightly to the mutilated bodies of his dead friends taunting him and blaming him for their deaths; later, he watches the man he loves get murdered, and the illusion includes spilled blood that does not disappear when the vision is over.
Second Apocalypse: The Prince of Nothing series has the Cants of Compulsion, a type of sorcery that allows the sorcerer to reprogram someone's beliefs and desires, completely altering their personality. It's a temporary effect, so everyone who undergoes it has to live with the trauma of having done things that they themselves would never do, even though they remember wanting to do it at the time. The only people who are immune to this effect are Mandate Schoolmen, since they already have an alternate personality living inside them.
In one Animorphs, Tobias is captured by an extremely sadistic Yeerk in the body of a young girl named Taylor and is subject to a torture that draws up his happiest memories and quickly swaps them with very painful ones.
In the second Sword of Truth, the ghost of Darken Rahl gave Kahlan a kiss in the neck, and she had a vision of being raped.
The Hitchhiker's Guide to the Galaxy has the Total Perspective Vortex, which shows the victim the vastness of the universe and how tiny and irrellevant he is in relation to it. It destroys the mind of anyone subjected to it except Zaphod, who really is the most important person in the universe because the universe where he meets it is a fake built specifically to save Zaphod from the real TPV.
In Robin Hobb's Farseer trilogy, The Pretender King and his agents use mind rape to forcefully help themselves to whatever a persons mind can give them. Be it information, loyalty, control of the body or simply for the sadistic pleasure of it.
Oddly enough, the conclusion of the trilogy saw Fitz use much the same technique back on "King" Regal, in a way that was not portrayed as even slightly anti-heroic. Then again, Fitz was the narrator, so...
A trainee Soul Drinker psyker in Crimson Tears uses his abilities on a human who died as a sacrificial combat slave under the lash of the Dark Eldar, and described it as "someone...someone tore out their souls."
Happens on a frequent basis in Ravenor, with even the title character using it, especially for a little Jack Bauer Interrogation Technique, and the trope name is used by someone who Ravenor involuntarily "wares" (basically wearing their body and mind like a glove). Meanwhile, "flects" are mind rape in narcotic form.
In the Anita Blake books, there's a few different kinds exhibited by the vampires. First, they use simple brainwashing of a human into a happy automaton with no independent thought (most often used to get people to stand still while they take blood). Not usually used for physical sex, however, taking blood this way is very sexual and is described as metaphysical sex. Then there's dream/magical simulation manipulation. While not used to its full potential, all the characters able to do this are nymphomaniacs and use it to force sex on the unwilling.
Thirdly, there's emotional manipulation - this can be either making people hopelessly in love with the vampire or making a person incapable of feeling anything but fear, and both types are shown both with and without physical rape. Finally, there's establishing the Human Servant/Master bond against the servant's will; basically, being a master vampire's human servant is the equivalent in being the wife in a medieval marriage, except the ceremony allows your husband to make you watch any memories he chooses during it and to play with your mind, to some extent, afterwards.
In the New Jedi Order series, teenage Jedi apprentice Tahiri Veila is kidnapped by Yuuzhan VongShapers, who attempt to rewrite her memories to convince her that she was a warrior of their species, as part of an attempt to create Jedi-fighting Force-sensitive Yuuzhan Vong. She's rescued before they finish, but there are still lingering consequences for the rest of the series.
In The Thrawn Trilogy, Joruus C'baoth goes far, far beyond the Jedi Mind Trick by mind-raping General Covell to death- reducing him to a state of such mindlessness that, when their link was broken by an Anti-Magic field, he didn't have enough mind left to survive.
Remember Star Wars: A New Hope, and the scene where Vader brings in an interrogation droid? In the novelization, and in the audio drama, he dismisses it in favor of what starts as a Mind Probe but, as she resists, quickly becomes this. It even gets to the point where he's able to make her believe that she is being burned alive, and that he is her father and needs to know what happened to the Death Star plans. Somehow, even this doesn't get her to talk. While she needs a medic after, she's also able to recover remarkably fast, shunting aside any effects just like all the other trauma she suffers in that film.
In Luke Skywalker and the Shadows of Mindor, the Big Bad, Cronal, wants to take Luke's body, but he has to soften up Luke's mind, first. He does that by forcing Luke to endure The Dark, an eternity of nothing but watching the stars go out. Luke's able to find a way out, but he couldn't take much more and for most of the rest of the book he is disturbed and nihilistic. Later, Cronal decides that Leia would be an even better choice, and when he does it to her he closes that avenue of escape, and cuts away her senses, her awareness of her body, and all of her memories. She takes much more than Luke did without breaking, and it's an open question whether she would have broken even without the interruption, though she too is in poor shape later.
Darth Zannah in the Rule of Two novel does this to a woman named Cynda who is holding her at gunpoint. She uses her powers to conjure up terrifying phantoms that only Cynda can see, and it is explicitly said that Zannah can stop it there, with Cynda only remembering the visions as a nightmare. But in the next moment an image of Cynda in bed with Zannah's now-deceased temporary love interest pops into her head, and Zannah pushes Cynda past the point of insanity, shredding her mind and leaving the tiny fragment of her consciousness that still exists irrecoverably trapped in torment. Later, Zannah pulls it on Jedi Sarro Xaj. He gets lucky however; Zannah just uses it to distract him so she can kill him.
Kyp Durron makes sure Qwi Xux can never build another Death Star again, by removing her knowledge of the Sun Crusher, so that no one can exploit potential weaknesses of his new toy. Wedge finds her weeping inconsolably in Kyp's wake, barely able to remember her own name, let alone what had just happened, but deeply traumatized by the experience.
In Timothy Zahn's Quadrail series the main Big Bad is a parasite that infects people's brains and brings them unknowingly into a group-mind. It can also take full control of the body at will once the infection is advanced enough.
There's a nasty one in The Algebraist by Iain M. Banks when police, who have been given permission to use whatever methods they think fit to break up a commune which is protesting against certain government policies, switch the total immersion virtual reality game the main character's girlfriend is playing for one which is described as "a nightmare of torture and rape" and leave her trapped in it for several days. she kills herself almost immediately after coming out of it and this event provides the main character's motivation for his betrayal of said government.
The practice for machine-life is so frighteningly simple to accomplish in Iain M. Banks's Culture novels, that the mental privacy of organic sentients is the closest that society gets to having a law. Any machine being found to be even reading a person's mind is given the apt title Meatfucker and is ostracized from the Culture. True to form for Banks, one of the protagonists of Excession is one such perpetrator.
...he constrained her (Niënor) to gaze into his eyes, and he laid a spell of utter darkness and forgetfulness upon her, so that she could remember nothing that had ever befallen her, nor her own name, nor the name of any other thing; and for many days she could neither hear, nor see, nor stir by her own will."
Morgoth's gaze is described as being able to drive even the lesser Ainur (aka gods) mad. Húrin, badass that he is, withstands it. Unfortunately, Morgoth then subjects him to a more mundane sort of Mind Rape, by cursing his children and forcing him to watch them destroy themselves.
In The Lord of the Rings, Sauron himself does this to Pippin through the Palantír. The movie portrayed it as poor Pippin screaming and thrashing around, unable to look away. Also, Frodo suffers similarly from the Ring, losing his ability to picture all that is good in the world, and only seeing himself as "naked before the wheel of fire."
In Heralds of Valdemar, Herald Talia is a sweet, benevolent person who nearly always uses her powerful empathic abilities to the benefit of all around her. The other .01 percent of the time, you learn why it doesn't pay to piss her off. The most dramatic (and literal) example was the time she forced a rapist to relive his victim's experience in a neverending loop, until and unless he could acknowledge that he'd done wrong.
The Edge Chronicles: Amberfuce is a master of Mind Rape. Telepathy is a rather common power among Deepwoods creatures, but Amberfuce possesses something most of them don't (or at least don't use): the power to erase, or rather freeze one's memories (and likely all emotions as well), which he uses to create perfectly obedient slaves. This is described as a painful process, compared to the touch of cold fingers inside your mind.
It is implied that this is what the Chamber of the Ordeal is like for those who wish to be knights in Tamora Pierce's Tortall Universe. The fact that it seems to be a semi-aware entity does not help. It shows you images of what can be, what you're afraid of, what you must do, and it acts as a portal for the God(s) words. If you're afraid of spiders and/or drowning, guess what you get visited by. People have described it has being a jewel, and the Chamber is a jeweler's hammer, tapping at your weaknesses. And those are the "good guys". If you're a "bad guy", or just not relevant to the plot, well... Squires have been known to kill themselves once they come out, come out dead, or come out feeling the pain of every injury they inflicted upon women that they beat, raped and murdered, and it doesn't end until it wants to. Anyone want to be a knight?
In The Godless World Trilogy it is strongly suggested this is what some of the na'kyrim were capable off before the War of the Tainted
Done to Slick Henry in the third book of William Gibson's Sprawl Trilogy as punishment for, of all things, car theft.
In Carrie, the title character does this to Sue as one final act of revenge for what she felt to be Sue's involvement in her humiliation at prom. Of course, Sue wasn't responsible for what happened — she let Carrie into her mind intentionally to prove this to her.
In Cordwainer Smith's short story "Mother Hitton's Littul Kittons" (read it here!), the Norstrillian defense system consists of amplifying and directing the psychic energies of specially bred insane minks in order to kill someone via mind rape.
The villain in Neuropath puts many people through this.
The eponymous evil cyborg from the future in David Weber's The Apocalypse Troll (one half of a Terminator Twosome) does this to several people in order to acquire information about present-day Earth. One victim is a schoolteacher who was camping with her husband; the "troll" has its way with her for five hours before finally disposing of her body, and records every moment of the process for its later enjoyment.
The three steps of the Aes Sedai test do this to whomever is taking them.
The Aes Sedai also consider bonding a Warder without said Warder's permission the equivalent of rape.
Compulsion, which is why it's banned. It's also first thing that most female (and we presume male) channellers try when they discover their powers - so even though the Weave itself has been lost to history, by piecing together the fragments the technique can be reasoned out.
Graendal in particular was known for using Compulsion, especially to make harems of beautiful men and women. In the Age of Legends they considered unrecoverable and it was considered mercy instead of murder to kill them - their minds are already gone.
Egwene mind-rapes one of the Chosen in Tel'Aran'Rhiod. The latter wakes up as a vegetable. Yikes.
The Dark One and his lieutenants to Chosen who fall out of favor. Don't fail the Big Guy.
The a'dam and its male equivalent mind-rape the wearer when they do something the leash-holder doesn't like. Other than their treatment of channellers - including the use of the a'dam - the Seanchan could be the good guys.
In the Star Trek: The Lost Era novel Well of Souls, the non-corporeal consciousness Uramtali forces her way into the minds of several characters, causing considerable damage.
This is step one of making a new vampire in Thirteen Bullets and its sequels. The vampire enters the victim's head and tears them apart mentally, hammering away until the victim is Driven to Suicide. That suicide turns the victim into a vampire. Arkley suffered this for a moment and is not sure if killing himself would cause him to come back as a vampire now; Caxton is brutalized for days this way, to the point it's a guarantee she'll turn if she ever does herself in, and then a piece of the vampire's mind got stuck in her head after she killed him.
Occurs with surprising frequency in Scott Westerfeld's Midnighters series, starting in the backstory when the 12-year-old Rex and Melissa shatter Rex's abusive father's mind, and going on through Madeline's meddling that enabled several of the midnighters to even be born, Melissa's forcible ripping-open of Dess's mind, and ending up with what Melissa, Rex and the darklings in Rex's head finally do to Madeline in Blue Noon. Fortunately, as the series goes on it's increasingly lampshaded that mindcasters are not nice people at all.
In David Weber's Path of the Fury, Commander George Rendlemann has knowledge of a conspiracy. Tisiphone (a refugee from the Greek Myths with minimal moral compass) rips the knowledge she needs from his mind via computer link. Instead of the catatonia you'd expect, he goes berserk.
In Trudi Canavan's The Black Magician Trilogy it's mentioned several times how horrible it is to have someone forcibly read your mind. At lot of people aren't too keen on truth reads (that you at least have to give your permission for and people can only see what you let them, how true that is is a different matter people seem to worry quite a lot if they have secrets as it will be hard to keep them hidden)
In Dean Koontz's False Memory, Dr. Ahriman is big on doing this. His Mind Rape of half the cast fuels most of the story's conflict, since it's difficult to know if what they're doing and thinking is really them and not his planted suggestions.
In Who Fears Death Onyesonwu finally gets fed up with the passivity of the citizens of Jwahir and literally does this to a crowd of them in the market square by forcing them to relive her mother's rape.
The Hunger Games: In Mockingjay, it turns out this is what happens to Peeta. In addition to other mental torture, the method the Capitol uses (called hijacking) is to revive good memories and while he's experiencing them, give him a chemical that induces intense fear and hallucinations, then let the memories restore themselves. The result is that, in addition to becoming an angry, paranoid and violent person, for some time afterward he has difficulties differentiating what is real and what isn't, and it's a while before he recovers.
Perryn of the Deverry books is capable of this. He knows that if he smiles and thinks in a certain way, women will be instantly attracted to him; he usually winds up sleeping with them. (He uses the same power to steal horses.) The real problems start when he decided Jill is the girl of his dreams, regardless of her being in a committed relationship with Rhodry. Perryn uses his power to make her go (and lie) with him, even though she specifically tells him to leave her alone (until the power simply overwhelms her mind). The kicker is that the power is a form of dweomer that not only slams Jill's own semi-latent abilities into overdrive, but the mere use of it is slowly killing Perryn. Part of the problem is that his current incarnation was born into the wrong race. (It's complicated.) It's no surprise that when Jill saw Perryn some time after that she was ready to kick the crap out of him.
Ixia and Sitia: In Magic Study, a character is accused of being a spy and subjected to this. She later states to her accuser that it is worse than rape and she would know.
In the prologue of the fifth book of A Song of Ice and Fire, Varamyr describes how skinchanging into human beings is considered an ultimate taboo, this in a culture that glorifies rape. He uses it as a last resort to live (and fails). Later in the book, we see that sweet, innocent Bran has picked up the habit of warging into Hodor and is unaware just how bad what he's doing really is.
This happens to Mercy Thompson, and is combined with genuine rape. The villain uses a magical fairy goblet to force her to do his will, including having sex with him and even loving him. It backfires on him, as he is wearing a sheepskin that prevents his enemies from hurting him. As at that moment, she loves him, it doesn't protect him from her, and she beats him to death. Two books later, she's still not fully recovered, although she notes that beating her rapist to death probably helped. She wonders if it will become a recognized therapy technique.
When one captured chimp sneeringly informs the aliens he won't tell them anything no matter what they do to him, they chillingly reply that cooperation isn't necessary... then strap him down in a machine that rips all the thoughts from his brain, painfully destroying both brain and body in the process. Yet he still gets he last laugh on them by making sure that his dying thoughts consist entirely of how much he despises them, and the irony of defeating their "undefeatable" machine.
The Gubru fuck with the minds of various chimps, without their remembering it, in an attempt to force the species to choose the Gubru as their next Uplift Stage Consorts. The victims only discover this under a similar psi-amplifying machine during the Gubru-financed Uplift Ceremony, when their minds are forced to obey the implanted suggestions against their will.
Legacy of the Dragokin: Abyss uses Psychic Powers to invade her target's mind, find their worst memory, and uses it to break them psychologically.
In the Foundation series novel Second Foundation, the First Foundation (scientific) creates a device which causes intense pain in those who have active telepathic abilities (such as the Second Foundation).
In Dune's Butlerian Jihad Trilogy the cymeks take brains from their human bodies (literal mind rape?), stick them in jars and turn the "thoughtrode" settings to make the minds feel pain. And then they are left on a shelf in their own little silent hell... for centuries.
In J. R. R. Tolkien's The Silmarillion, Túrin Turambar and his sister Niënor get cursed by the Big Bad and go through a Trauma Conga Line. The thing is, when it all started, Túrin was all of eight and Niënor hadn't even been born yet — the person the Big Badreally wanted to break was their father, who had to sit there for twenty-seven years and helplessly watch it happen. It worked. See also the previous category for a less mundane example in the same story.
Harry Potter: The Cruciatus Curse causes excruciating physical pain. However, extended use of the curse causes complete insanity in the victim; poor Neville's parents got to experience it first-hand.
In The Republic Of Trees, Isobel tried to leave the group, which by the laws of the Republic is punishable by death. The only alternative to death is "correction therapy" - unfortunately, with each of the other characters at different stages of their own Sanity Slippage, the role of the therapist is left to Joy, the second in command in the group, a Knight Templar about the rules and, which others haven't realised yet, a newly self-discovered Yandere.
While he is dubiously successful, practicing at this seems to be Beineberg's hobby in The Confusions of Young Törless. We only get to see a fraction of what he does with Basini, but there's plenty of brutal psychological humiliation, dicking around with hypnosis and Cold-Blooded Torture, as well as physical rape.
The tracker-jackers in The Hunger Games cause hallucinations of your worst fears. As we find out later, the Capital has been playing with the venom—low doses, carefully applied, can teach you to associate any particular thing with fear and horror.
In the web-novel Domina, Malcanthet, the "Queen of the Succubi," used mundane rape to break people and make them her slaves before the series started.
In A Song of Ice and Fire, Ramsay Bolton tortures Theon Greyjoy to the point where he forgets his own name, and is convinced he is Ramsay's loyal servant Reek.