in Quest for Fire, the protagonists come from the Oulhamr, a warlike and misogynistic horde of neanderthals. Their enemies are cannibals and other hordes who try to kill them on sight for trespassing but the narrative makes it clear that they would react the same way to strange men on their territory.
In Super Minion, the heroes and the protagonist's group of villains might not be perfect, but there are definitely far worse people out there.
Worm runs on this trope. The heroes working for the government are supposed to be the good guys, while their opponents include two different race gangs, a drug-dealing sex trafficking mob, a man who almost succeeds in taking over the city, a band of mass-murdering psychopaths, and three godzilla-tier monsters that are slowly destroying the planet. On the other hand, the heroes suffer from arrogance, in-fighting, and bureaucracy, and it's revealed that many of them got their powers illegitimately, from a corporation that used insanely unethical human experimentation to create them.
J. R. R. Tolkien's The Silmarillion. On one side are Elves and Men, the assumed good guys, who are flawed, cocky, haughty, corrupt, petty, jealous and well capable of killing their own kin, sabotaging their own cause; on the other side is the literal Dark LordMorgoth, who is a being of pure evil. (And Ungoliant, though she doesn't really care for any side and just wants to eat everything.)
The Johannes Cabal series features this throughout as fits its dark, but comic, tone-Cabal himself is a 'good' man in the thinnest sense, willing to go to any lengths to get what he wants-which is to reverse death completely. He appears as a hero because his main foes are always worse than him-Satan; a murderous backstabbing warmonger aristocrat; an Outer God; and a secret society with an army of monsters, among others.
The fourth book of the Chronicles of Nick has Ash state this outright. As an eleven thousand year old immortal he has seen quite a lot of good and bad, and basically tells Nick that everyone is capable of good and evil. The best you can hope for is to leave the earth a better place than you came into it.
Animorphs pitted six children against the Yeerk Empire, an expansionist and militaristic alien confederacy that occupies and enslaves Earth in secret. The main characters, all kids under the age of sixteen, are hopelessly outgunned and outnumbered and are pushed to using ever-more desperate and morally reprehensible tactics against an enemy that grows stronger no matter what they do. By the end of the series, the kids are just as ruthless as the people they fight. Can also cover the Andalite military, who are just as bad as the Yeerks, just in a different way.
R. Scott Bakker's Prince of Nothing trilogy exemplifies this trope. The most important character in the series, Anasurimbor Kellhus (there are several protagonists, but Kellhus is really the central character of the trilogy), is a ruthless, brilliant manipulator, part of an order of ascetics who have spent nearly two thousand years in isolation breeding and training for intellect, rationality, and the ability to "read" other people by their actions, mannerisms, and faces, thus "possessing" them and turning them to their will. Over the course of the trilogy, he comes to be seen as a Prophet, and eventually dominates the entire Three Seas area that composes the main setting for the books (he also comes to believe that he really is a Prophet). That sounds pretty horrible, until you remember that the primary antagonists, the Consult, are a cabal of human and non-human sorcerers and generals (including the Inchoroi, an alien race that fell into Earwa thousands of years before the books' story and who are defined by cruelty and an utter obsession with slaking their lust) seeking to resurrect a being that causes all children of races with souls — namely, humans — to be stillborn, so that they can drive the number of ensoulled beings in the world down below a certain number in order to prevent the certainty of their facing damnation and hell-fire upon their deaths. So yeah.
The civil war in Dread Empire's Fall; The "good side" is a massive, tyrannical empire that bombs worlds if they don't join them, and torture is an encouraged form of punishment.
The first book of Black Legion has the conflict between the budding Legion and the Emperor's Children. While Khayon and co kill people in droves, care little about the fate of slaves and aren't above a Mind Rape or two, they have standards. This... can't really be said about the Children.
Cat Girls Have Four Ears by Asi Hart: a budding alcoholic detective ends up working for a mad scientist engaging in human experimentation and murder in the fight against a villain bent on the destruction of civilization and subjugation of humanity.
Trainspotting: Almost all of the main characters are amoral drug addicts. The ones that aren't are either dead, going to be drug addicts in the near future, or berserker psychopaths. Or dead. Or are going to suffer because of the main characters.
And it's even more complicated than that. The book talks about how people who are going to be drug addicted are better before taking any drug: for instance, everyone says that the drug dealer was a nice man before taking heroin. It's more something like White-and-Grey Morality.
Joe Abercrombie's The First Law series is based on this principle, pushed to the point where you wonder at the end whether the protagonists were really the least evil, or if, perhaps, they weren't actually even worse than their antagonist.
Discworld's Vetinari sees the world in these terms, although the books themselves have genuinely good people.
After giving the page quote, Vetinari talks for about a page and a half about just how muchpeople suck, at which point Vimes asks him how he manages to get up in the morning, which he answers with his usual calm, kind-of-cheerful manner. However, Vetinari, who is not known for casually misspeaking, also says "there's a good man" about Vimes as Vimes exits after this speech.
Vetinari rules his own city, which is the most efficient city on the Discworld and there are people flocking to live there. Whether he's right or not, it works.
Petyr Baelish is an interesting case. Byronic as anything, quite highly sympathetic with a backstory tragic enough to break hearts and not without a solid point or too, sure. So, a Grey vs Gray case study in complexity, right? Nope: black as pitch once you get to know the scope of what he's been busy doing for years, let alone how and why. A very densely layered pit of pitch, tar, bitumen and crude with a touch of bonkers.
Played with and subverted in Glen Cook's The Black Company series. The soldiers work for an obvious Big Bad, and the rebels on the side of good turn out to be nasty little bastards. But every time it looks like the story's going down a familiar route, it ends up going somewhere even more interesting. In the end, the first book (The Black Company) ends up looking like a neutrally-portrayed reality while standard fantasy epics look like the propaganda put out after light's victory, and it gets more interesting from there.
The front-cover blurb for the first book reads, "Darkness wars with Darkness ... until the new Light breaks."
Tom Holt's Paint Your Dragon does this to the story of Saint George and the Dragon. Both are absolute assholes, but the dragon seems a little more sympathetic... Although considering he, at one point, annihilates an (occupied) theatre in an attempt to deal with George, this is more a statement on how unlikeable St. George is than anything else. The dragon's status as the Least Evil (?) character is cemented at the end, when the two end up switching forms and George's first action as a dragon is to kill the entire audience for their deathmatch in order to ensure that nobody with a rocket launcher is lurking in the stands).
The Dresden Files often works in this area. On more than one occasion there was no "good" solution so Harry often has to make do with what he can. An example in White Night occurs when Harry offers criminal Anti-Villain Marcone even more power to both get his aid and offer Chicago more protection against the supernatural.
It's also Lampshaded in Turn Coat when Harry dubs the clandestine group designed to combat the equally clandestine Black Council the "Gray Council." Oddly enough, they're probably less morally ambiguous than the stagnant, zealous, overly traditionalist leadership of the White Council.
The original Dune is very black-and-gray. The vast majority of the protagonists, including Paul, are not nice people and in many cases not good people either. And then there's Leto II in the sequels...
The first book doesn't possess this to too high a degree, but Paul and Jessica do patently manipulate the Fremen and their religion and lead the Fremen into ever-increasing acts of violence. The Corrinos are also a lot more gray compared to the pitch black Harkonnens.
Paul is definitely a lot more this trope in Dune Messiah, his jihad has killed billions and oppressed trillions, he himself compares himself unfavorably to Hitler but he does hate doing it and is only doing because he sees no alternative for the future of humanity. Then he learns in Children of Dune that he wasn't cruel enough!
Leto II's rule makes Paul's look like a paradise vacation, and while he is bothered by it, it's not nearly as much as his father was, because the Fremen as a people embody this trope and are trained from childhood to chose between two evils. What's the alternative to Leto's dark gray 'golden' path though? The complete extinction of humanity.
Brian Herbert and Kevin J Anderson's much-maligned Dune prequels actually do a fairly decent job portraying the free humans in terms of Grey morality. The Machines and their cyborg servants, on the other hand, are Card Carrying Villains. Although some are treated with some sympathy (especially in the last two books where the authors get better at making some of them like Erasmus actual three-dimensional characters), they're a bunch of bloodthirsty enslaving bastards who perform Mengele-style medical experiments on humans, get thrills from torturing them, force them to slave away like the Jews in The Ten Commandments apparently just because it strokes off their egos (little else makes sense, given that they can build sapient robots and contented humans would be less likely to rebel), and respond to any defiance with horrific atrocities. It's especially grating because superhuman machine intellects that run on cold logic should logically be Magnificent Bastards or at least dispassionate Chessmaster types, not a bunch of gratuitously sadistic Obviously Evil loons (in fairness, it's justified by one of the human Titans having programmed Omnius with his own personality).
In Dragaera, Vladimir Taltos is a low-level mafia boss, with all the unpleasantness that implies. However, he tries to be benevolent to his underlings and the inhabitants of the area he runs, and his antagonists are usually those causing or planning something that will cause widespread suffering. After leaving the Jhereg, while he tries to help the downtrodden, he does so through rather brutal methods.
This also applies to Vlad's friends Aliera and Morrolan. Both are ruthless and quite selfish, but are nicer to humans/arguably less of a danger to Dragaera than their fellow nobles. Thus, in Dragon, Vlad sarcastically notes the irony of calling Morrolan's army in which he is a member the "good guys", since all they are doing is trying to take some artifacts of doom/empathetic weapons so that a somewhat worse noble can't have them. Similarly, the plot of the novel, Iorich involves Vlad trying to defend Aliera after she is arrested on a charge of using illegal magic (the same type her father used and accidentally destroyed the old capitol and killed everyone there). This isn't because Aliera is innocent. Rather, it's because so many nobles break this law, that there must be a conspiracy at play for Aliera to be arrested for something she does in essentially plain sight.
Conan the Barbarian, especially Robert E. Howard's original stories. The hero is a mercenary/pirate/bandit/professional thief who isn't shy at any activity that involves cracking skulls and getting wenches and loot, but his personal code of honour means he doesn't kill wantonly and he always keeps his word. He's often pitted against fiendishly evil sorcerers, mad kings, or mad, fiendishly evil sorcerer kings. He gets a much more sympathetic portrayal when he becomes King of Aquilonia by his own hand, as he becomes a benevolent king who is defending his throne from scheming aristocrats and ambitious rivals.
J. K. Rowling was very fond indeed of doing this with her characters in the Harry Potter series to created rounded, complex characters.
Word of God says that there were concerted efforts made to remind the readers that Harry is a flawed person. Despite his good heart, Harry will resort to more extreme measures if necessary, such as his use of the the Cruciatus curse on Amycus, and before then, Bellatrix. As a result of the trials he goes through in the first four books, Harry experiences a long stretch of anger in Order of the Phoenix. Meanwhile, his best friends Ron and Hermione undergo similar characteristics. Ron, who never went through what Harry did but accomplished more than most Hogwarts students could ever admit to, decides to leave Harry and Hermione in the woods. Hermione enchants the D.A. list to give scarring acne to any member who betrays them and tricks Umbridge into going into the Forbidden Forest in the middle of the night, where Umbridge is attacked by centaurs.
Dumbledore, of whom so many people "thought the sun shone from every orifice", made plans in his youth with another to take siege of the general Muggle population, during which time he neglected his remaining family. In the last book, Dumbledore reveals that Harry would need to be sacrificed in order to defeat Voldemort in the end. Horrified, Snape states Dumbledore "kept [Harry] alive so that he can die at the right moment" as Snape had done everything he could to keep the son of his one love, Lily Potter, safe. Dumbledore explains that Harry's sacrifice is necessary for Voldemort's downfall.
Paradoxically, Regulus turns out to have been not as Black as first painted — same for Snape. Draco is a tricky one, who at first doesn't turn Harry in, but later tries to capture him, accompanied by his old henchmen who, by now, are not just brainless brawns and are unafraid to kill — though given his family's precarious situation, desperation could have been a factor.
Other characters such as James Potter and Sirius Black are shown to have good hearts as well, but could definitely be Jerkasses at times (Sirius and his treatment of Snape/Kreacher, his recklessness).
Some members of the Ministry of Magic like Rufus Scrimgeour are firmly on the gray side but are likewise still lighter than the black morality of Voldemort.
Cataclysmic Horizons contains the Sodality of Gerosha, a band of superheroes many of whom are only possible because their parents were subjected to crimes against nature. They consist of a horny and insecure girl with centipede powers, an impulsive and aggressive woman with a stolen Powered Armor, a sleazy womanizer who can jump incredible heights, an angry and vindictive plant-man who is himself a victim of crimes against nature, a sociopathic Reality Warper, a Robin Hood archetype, and more. Together, they fight against Those Wacky Hebbleskins, who are essentially A Nazi by Any Other Name. Who have their own army of evil freaks, who are also only possible because their parents were victims of crimes against nature.
Martha Wells's Death of the Necromancer has NicholasValiarde, a cold-blooded thief, murderer and all around Magnificent Bastard. Nic has spent years sabotaging his enemy on a Roaring Rampage of Revenge; at the start of the narrative, Nic's nearing the completion of his ultimate scheme when he and his subordinates run afoul of an unknown person using Black Magic. Somehow, this leads to the group spending the rest of the book fighting an insane mass murderer. And the reason they do it is at least partly because it's bad for business.
In Andrew Vachss's Burke books, Burke and his True Companions are mostly ex-cons who skirt or break the law frequently. They cross paths with pedophiles and other worse scum from time to time.
Near the end of Good Omens, the forces of Heaven and Hell line up across the sky, and the narrator mentions that if you looked very closely, and had been specifically trained, you could tell the difference.
Common in the works of China Mieville. Kraken, for instance, has a Lovecraftian doomsday cult as one of the nicer factions.
Tadeusz Borowski's Holocaust stories feature the occasional good character, but they don't tend to live long in the atmosphere of the camps. The characters who do survive (at least for a while) are those who're willing to steal from others, to betray each other to the guards, to help in the execution of the Jewish inmates, and to eat the corpses of their fellow prisoners so as to avoid starvation.
The Tribulation Force versus the Global Community in the Left Behind books.
In one of the books, a minor side character calls the Trib Forcers on this. Another minor character turns out to be a con man, and the Tribbers reverse the con to screw him over. And just as the main characters (and, likely, the reader) are thinking "Yeah; he finally got what was coming to him" the first minor character says something like "He's been pretending to be saved, but he's not, so he's going to Hell, forever; that wasn't enough for you so you felt the need to cheat him out of a few thousand dollars too? You're supposed to be better than he is. I don't think I can work with you guys any more."
The Mental State takes place mostly in prison. Inevitably, most of the characters are criminals. The central character spends most of his time trying to separate these characters into Grey ones and Black ones. He himself is a ruthless and ambitious young sociopath whose actions simply happen to benefit the people with Grey morality and punish the ones with Black morality. To put things into perspective, the kindest, friendliest and most loyal inmate is a closet lolicon.
Sisterhood Series by Fern Michaels: As the series goes on, the morality of the stories turns into this. The good guys are called the Vigilantes because they break the law in capturing a bad guy and inflicting a cruel and unusual punishment on him or her. The good guys don't kill anybody, but since their punishments tend to be of the Fate Worse than Death variety, that fact may not be very comforting. Also, the good guys have acted like big-time Jerkasses a number of times. That's okay, because the bad guys have virtually no redeeming qualities to speak of!
Percy Jackson and the Olympians: Help the Gods who are often jerkasses and sometimes cause problems, or serve a Titan who devoured his own kids and uses humanity as a source of cheap amusement or as a snack. It's Lampshaded by the main characters, other demigods, and even some of the gods themselves that while the current system is by no means perfect, it's still much better (in that the gods legitimately enjoy the modern world more or less) than what the Titans and the other primordials would want out of the world (usually starting with the destruction of human civilization as we know it).
A Series of Unfortunate Events, especially from book eight onwards. While Olaf and the fire-starters are unambiguously evil, the "good" side of VFD isn't all sunshine and rainbows either, as the Baudelaires discover. The Baudelaires themselves also commit "bad" actions due to circumstance.
In the Indian novel The White Tiger, some people (like Balram and Mr. Ashok) have their Pet the Dog moments, but it's pretty clear everybody else kind of sucks.
And Then There Were None: All of the characters brought to the island are guilty of very heinous crimes, with some being more sympathetic than others. They are being stalked by someone who wants to punish them for their crimes. The real killer, Justice Wargrave, was himself motivated by a combination of sadism and justice to commit his crimes.
The Adversary Cycle: In The Keep, it's Rasalom vs. Nazis. Guess who's worse. Also, the Otherness vs. the Ally in general. The Ally isn't evil for the sake of being evil, but it certainly does horrible things. However, Earth needs the Ally to win because the Otherness would be much, much worse.
In Zoo City Zizi December is a cynical ex-drug addict who blames herself for her brothers death. Everyone else is as bad or worse, sometimes much worse.
In Tom Kratman's Caliphate, the Imperial States of America isn't exactly a shining "white hat", but as presented in the novel it's a better option than the Caliphates (particularly the one on which the novel is focused), who are very much of the "black hat" persuasion. The other nations that get any attention aren't much better than the ISA.
In The Leeshore by Robert Reed, the Alteretics are a faction of humanity that worship an artificial god. They get "conscripts" by capturing enemy soldiers, hooking up to "glass wires" which are used to make them fanatically devoted to the priests and the artificial god. Their first act was to start bombing every manufacturing center they could find in the Solar System, and population centers as well. When they fled the Solar System towards the tiny outpost on Leeshore, the first thing they did upon landing was to kill everyone they could find, to try and prevent the inhabitants from cutting loose their Space Elevator. The Asian Alliance that is chasing the Alteretics down use the same "wires" to control the emotions of their own soldiers by reinforcing certain emotions and thought patterns, at the cost of making them extremely bloodthirsty. The two protagonists are captured by the alliance are forced into assisting them in hunting the Alteretics.
The designated heroes of The Chronicles of Magravandias are not particularly heroic, something which they and the antagonists deconstruct over the course of the trilogy. In the end, it's not that the protagonists are good so much as the villains of the story are worse.
The impetus of The Mortal Instruments series is Valentine Morgenstern's belief that all Downworlders should be exterminated, despite Shadowhunter laws that protect him. Pretty evil right? Then you learn that despite these supposed protections, prejudice against Downworlders is pretty heavily ingrained into Shadowhunter culture. The issue is lampshaded several times throughout the series, but it comes to a head at the end of City of Heavenly Fire when the Clave begins passing legislature inhibiting the freedom of faeries, despite Magnus' warnings that it will only cause future problems. If that wasn't enough, they also begin discriminating against Shadowhunters with faerie blood, abandoning one who was kidnapped by faeries and risked his life to leak vital intelligence. The author herself says that was intended to make the readers realize that the Clave isn't just flawed and in need of a few tweaks, but crippled by its own prejudice.
The Daniel Faust series: Daniel's crew includes criminals, sorcerers, and a demon or two, but the monsters they go up against are even worse.
In Humane Tyranny certain members of the rebellion believe this to be a necessary evil and that in order to stop the government from killing innocent people, they must perform evil deeds that will damn their very souls. The main characters, though, wish to avoid this trope entirely and to do things the right way.
Shattered Continent: Chapter One of Caroline's Awakening features a team of mercenaries having a frank discussion about using fire or poison gas against an enemy structure that may or may not contain innocent captives. They're the protagonists. The opposing force is a bunch of half-undead, demon worshipping cultists and there are no hostages because the cultists ate them all already.
WASP: The main character is a military secret agent working to disrupt the enemy government. Some of his actions are morally gray, such as when he booby-traps two civilian merchant ships, implicitly causing their crews' deaths. On the other hand, the enemy is a militaristic and quite oppressive empire, and their State Sec agents we meet in the book are pretty much all ruthless and brutal.
Revanche Cycle: In a story that's essentially about the price of power, even the characters with genuinely good intentions end up with blood on their hands (and several of the villains have perfectly reasonable motives for the heinous deeds they commit).
On the one hand, you have armies and gods of Eldrich Abominations who murder, torture, and worse For the Evulz. On the other, you have a madman who kills his kin, relies on his victims' souls for his vitality, and actually worships the monsters he fights. Ladies and gentlemen, The Elric Saga.
The Man in the High Castle: On one side of the world is the Empire of Japan, which remains a brutally oppressive dystopia, but they run their colonies and puppet states in a fairly negligent, liberalized way. On the other is the Greater Nazi Reich, who are borderline omnicidal maniacs driven by their war to attain racial purity by any means necessary.
The Ultimate Killing Game would be Evil vs. Evil if not for the Feds. Without them it would be a story in which a strange cannibal cult takes drugs to cause apathy to make it easier for them to do their thing, and some really vicious criminals with no regard to human life. The feds may be prone to crime, but at least they're not evil.
In Victoria, set in a Fallout-like post-apocalyptic America, nearly all of the various successor states are some flavor of politically Ax-Crazy, ranging from human-sacrificing neo-Pagans and genocidally misandristic high-tech Amazons to cannibalistic Mexican warlords and unapologetic Nazi imitators. Victoria itself is fascist, and would be the villains in any other story, but given the crapsack nature of the setting and the author's political leanings, they're the good guys.
A running theme in the The Death Gate Cycle. Of the two main factions, the Patryns are conquest-crazy Blood Knights and the Sartan are paternalistic and hubristic Neglectful Precursors. Both of them treat humans, elves, and dwarves like disposable toys — not that the humans, elves, or dwarves are much better. Then the pure evil dragon-snakes show up trying to induce universal suffering so that they can feed on it, inspiring the relatively more moral of the first groups to band together against them.
Han and his Caper Crew are smugglers, thieves, and con artists — low-level criminals just out to make a living. The goal for most of them if they make the big score is to pay off their debts and retire. Their target is a high-ranking member of the galaxy's most notorious Syndicate, Black Sun, who regularly engages in high-level blackmail, arms dealing, and murder, and the money they want to steal from him was stolen from one of them in the first place.
Dayja is an Imperial Intelligence agent, working for the oppressive totalitarian Empire. He has very few qualms using a frequently lethal Truth Serum/Laser-Guided Amnesia drug in an interrogation. But his mission is to bring down Black Sun, and to that end he surreptitiously aids Han's crew in order to get a set of blackmail files that are causing the Empire a great deal of trouble.
You can count the number of genuinely heroic characters in Market of Monsters on one hand. Even the deuteragonists commit multiple acts of murder and torture as a result of a lifetime of abuse and living in a world where they're declared monsters by nature and have decided Then Let Me Be Evil.
The Arts of Dark and Light takes Deliberate Values Dissonance seriously, so none of the factions in its Medieval European Fantasy looks completely "white" by modern standards. Amorr, probably the closest thing this world has to a "Good Guy" faction, has social values largely appropriate to ancient or medieval times, and so (for example) slavery is commonly accepted there. However, their enemies are far worse, with Savondir a totalitarian monarchy that combines the worst features of medieval despotic feudalism and early modern French Revolution-style tyranny. And even they are better than the pure destructiveness of the Hordes in the North.