The Nightbringer, an Omnicidal ManiacPhysical God that usually takes the form of a forty-foot-tall, flying, metal Grim Reaper is reputed to have, at the dawn of time, Mind Raped proto-life so comprehensively that he instilled the fear of death in all living creatures in the galaxy (except the Orkz). He may have also created entire races just to feed on their fear.
Eldar Farseers can have a psychic ability called 'Mind War,' essentially a Mind Rape as a weapon to burn out an enemy's brain and kill them.
Since Psychic Powers are drawn from a hell-dimension Psykers face a lifetime with the threat of suddenly being attacked at any moment. For the lucky ones, Mind Rape. For the unlucky ones, Demonic Possession.
The training of a psyker probably involves preventative Mind Rape, as evidenced by the "sanctioning side effects" table in Dark Heresy. Among the results are: eyes burned out, white hair and gibbering, hair loss, chanting scripture in your sleep, visibly grimacing whenever you hear mention of Holy Terra, and believing that parts of your personality that were forcibly removed have gained sentience and are tracking you down.
The process of creating an astropath involves a normal human psyker making psychic contact with the Emperor for a brief instant. The process is so traumatic that it burns out the subject's eyes.
Inquisitors frequently use this one. Inquisitor Ravenor is particularly adept at this.
What the Emperor uses on Horus, destroying his soul, which was some really nasty business — ending the Horus Heresy.
The psycho/hypnotherapy Space Marines undergo as part of their conversion from human to Astartes is a limited form of Mind Rape, a sort of mental The Spartan Way. Grey Knight training takes this even further, involving as it does the "Six Hundred and Sixty-Six Rites of the Emperor", which is essentially longhand for 666 mind rapes.
A spell called Mind Rape appears somewhere in the Dungeons and Dragons "Book of Vile Darkness" sourcebook. It lets you completely rewrite or erase the victim's memories, feelings, and alignment. Naturally, it has an [evil] tag, which is D&D's way of marking a spell as, well, evil... amusingly, there is another spell, Programmed Amnesia, that does nearly the exact same thing with no evil tag. Presumably it's all about the name, or maybe just how you use it.
The implication is that with the Mind Rape spell is something that a) hurts a lot and b) is actually forcibly removing the memories and character traits. Selective Amnesia is a subtle blocking of certain parts. Both are a violation, but the implication is that the Amnesia is supposed to be used for blocking traumatic experiences and such.
Another spell exists in the divination category called Terrible Secret, which causes the caster to reveal a mind-shattering secret to his opponent. The secret is so horrifying it causes the creatures brain to simply malfunction and potentially. It can also be applied to a group with the upgraded Terrible Revelation.
The Mindflayers (Illithids) have psionic powers that can mind rape a character; possible effects include permanent insanity, rage, confusion, coma, and death. The worst turns a victim into the illithid's "thrall", or complete slave, a state that can usually only be undone by a third party killing the illithid.
In 4th Edition, there is a "psychic" damage type, implied to be exactly this. It can kill people.
In the Pathfindersupplement Ultimate Magic, there is a whole raft of these sort of spells. They range from Murderous Command which is exactly what it sounds like (you order someone to kill the person closest to them), Malicious Spite (make someone hate another person for days and work to harm them constantly), and the granddaddy of them all, Prediction of Failure, which forces you to experience the pain and grief of every single failure and mistake you will ever make in your lifeall at once. FOREVER.
Mage: The Awakening has a spell called "Psychic Violation" which essentially does this to people. The effects include sapping their will, potentially driving them insane, and giving them a pathological need to avoid confronting the caster. There is another spell, "Nightmare Journey", which takes the concept of Mind Rape a step further by detaching the subject's consciousness, and projecting it into the mind of an abomination. Both spells are mostly practiced by a group of mages whose whole creed essentially revolves around Mind Rape, and can only be performed by a person with a criminal mentality without potentially putting a ding in the Karma Meter.
There is also "Dislodge the Soul", a spell that allows one to mess around with a person's soul so that they lose all feeling of connection or empathy for other humans (this is represented in game terms by automatically failing any roll to prevent a ding to the Karma Meter), an experience that can be intensely disturbing for those who experience it. The only way to recover (without magic) is to engage in activities that reaffirm one's connection to humanity (such as a parent playing with their children); this naturally becomes more difficult as Morality decreases. It is also possible for the casting mage to take a glimpse at what lies behind the damaged soul. Most find it incredibly disturbing; the Echo Walkers (who invented the spell) find it inspirational, as they believe it lets them see the angels they want to emulate.
Subverted with the acamoth. Sure, they're evil, reality-hungry spirits who recreate their horrific home in the minds of those they enter...But it was entirely consensual, to the point that whatever they do with their host dings the Karma Meter, since you literally let them in, and are thus a willing accomplice to their deeds...
For the Banishers, the Awakening (the moment of becoming a mage) itself is Mind Rape. The point is particularly driven home because it is usually a profound and joyous, inspiring moment (even in some of the less pleasant places, like Pandemonium or Stygia). To Banishers, it is unwanted, misunderstood, or traumatic, in a way that causes them to want to destroy all magic.
Two vampire clans have this as a power. In the Vampire: The Masquerade, Malkavians used Dementation to drive potential victims and rivals insane. In Vampire: The Requiem, Nosferatu use Nightmare to inspire great fear... and break minds with it.
Mind Rape tends to be what changelings go through during their stay in Arcadia. Notably, the driving ethos of many of the changeling Courts seem akin to the symptoms of post-traumatic stress disorder.
The Earthbound of Demon: The Fallen have access to the Lore of Violations. Its second rank is called "Mind Rape," which allows an Earthbound to rummage through a mortal's thoughts and memories, causing physical damage in the process.
Due to the metagame explanation for Magic: The Gathering (basically, two almost-all-powerful wizards fighting, with cards representing memorized spells) most spells forcing players to discard cards are flavored as Mind Rape. These cards tend to be blue (the color of the mind, mind control and trickery, among other things), black (the color of corruption, insanity and power at all costs, among other things) or both.casesinpoint.
The flavor explanation is that a discard spell functions by reaching into the enemy mage's mind and destroying their knowledge of particular spells before they can be cast. There's an example in the Ice Age block novelisations where the protagonist, Archmage Jodah, engages in a battle with an evil wizard. He gains an advantage by using mass-discard spells to tear apart his opponent's mental library of spells.
And there's an in-story example in Agents of Artifice, where Jace Beleren does this to Tezzeret after winning a duel against him, in a rare hero-to-villain example. Represented in-game by the card Mind Sculpt.
On a different line of logic, if a player cannot draw a card because his/her library (Magic-speak for a player's deck while in a game) is empty, he/she immediately loses, described in-universe as that planeswalker going insane and being unable to continue fighting. There are some spells like Traumatize and Glimpse the Unthinkable that put cards directly from your opponent's library into their graveyard, which tend to have this type of theme. The original Millstone◊ is essentially magically-aided brain torture through loud and repetitive noises.
The Nemesis of Reason forces the opponent to discard ten cards from his or her library each time it attacks. In other words, it is so horrifyingly wrong that it erases one sixth of your mind (standard decks consist of about 60 cards) when it looks at you funny.
In the novel Planeswalker, Gix does this to Xantcha with frightening ease, leading her to muse that his name must be the Phyrexian word for "rape".
Exalted has quite a lot of mindrape powers, most spectacular being the Border of Kaleidoscopic Logic martial arts style, which doesn't only fundamentally and permanently rewrite one's mind, but also does things like permanently locking the target in an illusion of being a perfect, flawless being, or denying the target the capability to comprehend any spoken or written language, ever.
Oh, there is one that beats this by several orders of magnitude: the process of becoming an akuma. The mindrape part, which goes on for hours, is bad enough. The next part, the soul rape, goes on for days. At the end of it, your personality and memories have been hacked up with a rusty cleaver and put together in a variety of horrible fashions (such as your happy childhood being converted into fifteen years of hellish abuse), and your former Motivation has been violently ripped out and replaced with an Urge, essentially a command that you can't gainsay (such as to end the worship of Ahlat or corrupt the Dragon-Blooded with demonic taint). And it gets worse. If you complete your Urge, you have to go through the whole thing again in order to get a new one. (And if your Urge becomes impossible, such as if you're programmed to kill someone and he gets hit by a stray Death of Obsidian Butterflies, you almost invariably go mad.) Unsurprisingly, the Yozis take care not to advertise that this is what happens when you sign up to their "get more power by serving the Yozis" deal.
There are a good number of Charms that do this, but it's no surprise that the most detailed of them emerge from the Ebon Dragon. Some of these fun tricks include Golden Years Tarnished Black (which target a beloved memory and paint it in the worst possible light), Want Becomes Need (which targets any memory and turns it into an almost fetishistic compulsion for the target), and Everything Gets Worse (which targets hated memories and makes them all that much worse).
GURPS has a number of ways to do this. The Terror advantage can cause permanent insanity and even reduce intelligence at high enough level. The spell Fear, Panic, Terror, Madness and Nightmare all can cause this.
GURPSDiscworld brings in the Break Mental Walls spell, which strips away the barriers people erect to deal with all the things about themselves they don't want to acknowledge; while it's in effect, the character is largely incapacitated and has to make a Fright Check every ten minutes. The more unpleasant mental traits the character has and cruel things they've done recently, the longer the spell lasts. It doesn't work, or doesn't work entirely, on three categories of people: people who're truly saintly, people who already know and like themselves, and complete psychopaths.
The "insanity beam" from Ultra-Tech first hits the target with incapacitating hallucinations of terror, then he drops into a coma where the horror continues unabated. If he survives that the nightmares continue for a few weeks afterward.
Infernum is full of this. First, you have powers like "Eat the Mind" (you literally tear apart and consume an opponent's mind through a psychic link), or "Nightmare Form" (not only can you become the living embodiment of a creature's worst fears, you become so convincing an embodiment you can kill them through sheer terror). Then, you have more subtle attacks... Imps can shrink down to such a tiny size they can crawl in through your ear and take control of your body by physically manipulating your brain. Malcubi can physically enter your dream if they touch you. Many demons have the power to possess you, either simply riding around as a spirit in your head or outright using your body for themselves. The kicker? These are your player characters, and you're encouraged to do this to your enemies.
Scion: The Justice Purview has this as part of its schtick. Everything from making the target suffer the effects of a crime they committed, to haunting them with the ghosts of their victims, to making them suffer decades of imprisonment, in their minds, in the space of minutes of real time. And most of these powers result in long lasting damage to the target.
Deadlands, the only game that has a space in the character's sheet for "your Worst nightmare.". Believe it or not, this description is actually functional to the game.
Some of the racist armor in FATAL screws with the wearer's character.
In Mutants & Masterminds, the most expensive version of the Mental Transform power can reprogram memories, mental traits, and personality. Additionally, the optional dreamscape/soulscape combat rules introduced in the Mecha & Manga supplement allow one to implant suggestions, simulate sleep deprivation...orworse.
In Feng Shui's dystopian 2056 juncture, the Bureau of Happiness and Productivity are one of the most horrific agencies of the Buro. Apart from making happiness drugs for the populace, they also specialize in breaking people through drugs and psychological torture to create the sociopathic slaves known as Bonechills.