While not a videogame, it's an inherent risk of running neutral or evil parties in Dungeons & Dragons. In the case of evil parties, this is arguably the point.
Some spells in the expansions seem made for this trope. Great examples are Inner Fire (cause target to slowly combust inside and out), Drown (cause target to drown on dry land by filling their lungs with magically respawning water), Choke (basically what Darth Vader does), and the Distort Limb and Distort Body spells (turn target into Body Horror).
Some editions of the game had "Heat Metal" as one of the standard, go-to cleric combat spells. It was a spell that caused metal to gradually heat up over the course of several rounds until it burned flesh on contact. The spell description specifically mentioned how excrutiatingly painful it was to try to hold on to something affected by this spell to discourage DMs and good role-players from simply deciding to hang on to their white-hot weapon if they had enough hit points to soak the damage the spell did. Unpleasant when used on your weapon, but remember: the spell was intended to be used against peope wearing metal armor. Heavier armors actually took longer to remove than it took the spell to heat them to flesh-melting temperatures.
The core rule books give you multiple spells each for controlling people's minds and influencing their perceptions to make them see whatever you want them to see (at least 3.5 and back gives you easy access to these spells). The ways in which you could screw with people with these spells are limited only by your imagination. And if your target is too strong willed to be influenced directly, just enchant weaker minded npcs to mess with them.
In 3.5 edition many spells note that they can only be cast on a 'willing' target, preventing them from being used on enemies for creatively horrifying purposes. However, a throwaway paragraph in the Player's Handbook notes that a helpless creature (such as one that is paralyzed or tied down and unable to move) is automatically considered to be 'willing'...
Earthdawn has its moments as well, at least as a Nethermancer. You get to skin people alive just by touching them, or breaking bones just by snapping your fingers or just inflicting pain (this one without physical injuries though) for the hell of it.
You would be astounded at the depths of malevolence various Charms in Exalted can reach. The Infernal Exalted have particular talent when it comes to such Charms; whether you prefer the direct sadism of inflicting a lingering death from magical radiation sickness, the elaborate torture of granting someone's wish and then demanding the head of their only child as payment (with horrible penalties if they renege on the deal), or the psychological angle of meticulously ruining all of some poor bastard's most precious memories in chronological order, there's an Infernal charm that has your back.