J. R. R. Tolkien's The Silmarillion. On one side are Elves and Men, the assumed good guys, who are flawed, cocky, haughty, corrupt, petty, jealous and well capable of killing their own kin, sabotaging their own cause; on the other side is Morgoth, who is a being of pure evil. (And Ungoliant, though she doesn't really care for any side and just wants to eat everything.)
The Johannes Cabal series features this throughout as fits its dark, but comic, tone-Cabal himself is a 'good' man in the thinnest sense, willing to go to any lengths to get what he wants-which is to reverse death completely. He appears as a hero because his main foes are always worse than him-Satan; a murderous backstabbing warmonger aristocrat; an Outer God; and a secret society with an army of monsters, among others.
The fourth book of the Chronicles Of Nick has Ash state this outright. As an eleven thousand year old immortal he has seen quite a lot of good and bad, and basically tells Nick that everyone is capable of good and evil. The best you can hope for is to leave the earth a better place than you came into it.
Animorphs pitted six children against the Yeerk Empire, a expansionist and militaristic alien confederacy that occupies and enslaves Earth in secret. The main characters, all kids under the age of sixteen, are hopelessly outgunned and outnumbered, and are pushed to using ever-more desperate and morally reprehensible tactics against an enemy that grows stronger no matter what they do. By the end of the series, the kids are just as ruthless as the people they fight. Can also cover the Andalite military, who are just as bad as the Yeerks, just in a different way.
R. Scott Bakker's Prince of Nothing trilogy exemplifies this trope. The most important character in the series, Anasurimbor Kellhus (there are several protagonists, but Kellhus is really the central character of the trilogy), is a ruthless, brilliant manipulator, part of an order of ascetics who have spent nearly two thousand years in isolation breeding and training for intellect, rationality, and the ability to "read" other people by their actions, mannerisms, and faces, thus "possessing" them and turning them to their will. Over the course of the trilogy, he comes to be seen as a Prophet, and eventually dominates the entire Three Seas area that composes the main setting for the books (he also comes to believe that he really is a Prophet). That sounds pretty horrible, until you remember that the primary antagonists, the Consult, are a cabal of human and non-human sorcerers and generals (including the Inchoroi, an alien race that fell into Earwa thousands of years before the books' story and who are defined by cruelty and an utter obsession with slaking their lust) seeking to resurrect a being that causes all children of races with souls - namely, humans - to be stillborn, so that they can drive the number of ensoulled beings in the world down below a certain number in order to prevent the certainty of their facing damnation and hell-fire upon their deaths. So Yeah.
The civil war in Dread Empire's Fall; The "good side" is a massive, tyrannical empire that bombs worlds if they don't join them, and torture is an encouraged form of punishment.
The first book of Black Legion has the conflict between the budding Legion and the Emperor's Children. While Khayon and co kill people in droves, care little about the fate of slaves and aren't above a Mind Rape or two, they have standards. This... can't really be said about the Children.
Trainspotting — Almost all of the main characters are amoral drug addicts. The ones that aren't are either dead, going to be drug addicts in the near future, or berserker psychopaths. Or dead. Or are going to suffer because of the main characters.
And it's even more complicated than that. The book talks about how people who are going to be drug addicted are better before taking any drug: for instance, everyone says that the drug dealer was a nice man before taking heroin. It's more something like "white and gray morality".
Joe Abercrombie's The First Law series is based on this principle, pushed to the point where you wonder at the end whether the protagonists were really the least evil, or if, perhaps, they weren't actually even worse than their antagonist.
Discworld's Vetinari sees the world in these terms, although the books themselves have genuinely good people.
It's worth noting that after giving the page quote, Vetinari talks for about a page and a half about just how muchpeople suck, at which point Vimes asks him how he manages to get up in the morning, which he answers with his usual calm, kind-of-cheerful manner.
Also worth noting is that Vetinari, who is not known for casually misspeaking, says "there's a good man" about Vimes as Vimes exits after this speech.
Also Vetinari rules his own city, which is the most efficient city on the Discworld and has people flocking to live there. Whether he's right or not, it works.
Played with and subverted in Glen Cook's The Black Company series. The soldiers work for an obvious Big Bad, and the rebels on the side of good turn out to be nasty little bastards. But every time it looks like the story's going down a familiar route, it ends up going somewhere even more interesting. In the end, the first book (The Black Company) ends up looking like a neutrally-portrayed reality while standard fantasy epics look like the propaganda put out after light's victory, and it gets more interesting from there.
The front-cover blurb for the first book reads, "Darkness wars with Darkness ... until the new Light breaks."
Tom Holt's Paint Your Dragon does this to the story of Saint George and the Dragon. Both are absolute assholes, but the dragon seems a little more sympathetic...although considering he at one point annihilates a (occupied) theatre in an attempt to deal with George, this is more a statement on how unlikeable St. George is than anything else. The dragon's status as the Least Evil? character is cemented at the end, when the two end up switching forms and George's first action as a dragon is to kill the entire audience for their deathmatch in order to ensure that nobody with a rocket launcher is lurking in the stands).
The Dresden Files often works in this area. On more than one occasion there was no "good" solution so Harry often has to make do with what he can. An example in White Night occurs when Harry offers criminal Anti-Villain Marcone even more power to both get his aid and offer Chicago more protection against the supernatural.
It's also Lampshaded in Turn Coat, when Harry dubs the clandestine group designed to combat the equally clandestine Black Council the "Gray Council." Oddly enough, they're probably less morally ambiguous than the stagnant, zealous, overly traditionalist leadership of the White Council.
The original Dune is very black-and-gray. The vast majority of the protagonists, including Paul, are not nice people and in many cases not good people either. And then there's Leto II in the sequels...
The first book doesn't possess this to too high a degree, but Paul and Jessica do patently manipulate the Fremen and their religion and lead the Fremen into ever increasing acts of violence. The Corrinos are also a lot more gray compared to the pitch black Harkonnens.
Paul is definitely a lot more this trope in Dune Messiah, his jihad has killed billions and oppressed trillions, he himself compares himself unfavorably to Hitler but he does hate doing it and is only doing because he sees no alternative for the future of humanity. Then he learns in Children of Dune that he wasn't cruel enough!
Leto II's rule makes Paul's look like a paradise vacation, and while he is bothered by it, it's not nearly as much as his father was, because the Fremen as a people embody this trope and are trained from childhood to chose between two evils. What's the alternative to Leto's dark gray 'golden' path though? The complete extinction of humanity.
Brian Herbert and Kevin J Anderson's much-maligned Dune prequels actually do a fairly decent job portraying the free humans in terms of Grey morality. The Machines and their cyborg servants on the other hand are Card Carrying Villains. Although some are treated with some sympathy (especially in the last two books where the authors get better at making some of them like Erasmus actual three-dimensional characters), they're a bunch of bloodthirsty enslaving bastards who perform Mengele-style medical experiments on humans, get thrills from torturing them, force them to slave away like the Jews in The Ten Commandments apparently just because it strokes off their egos (little else makes sense, given that they can build sapient robots and contented humans would be less likely to rebel), and respond to any defiance with horrific atrocities. It's especially grating because superhuman machine intellects that run on cold logic should logically be Magnificent Bastards or at least dispassionate Chessmaster types, not a bunch of gratuitously sadistic Obviously Evil loons (in fairness, it's justified by one of the human Titans having programmed Omnius with his own personality).
In Dragaera, Vladimir Taltos is a low-level mafia boss, with all the unpleasantness that implies. However, he tries to be benevolent to his underlings and the inhabitants of the area he runs, and his antagonists are usually those causing or planning something that will cause widespread suffering. After leaving the Jhereg, while he tries to help the downtrodden, he does so through rather brutal methods.
This also applies to Vlad's friends Aliera and Morrolan. Both are ruthless and quite selfish, but are nicer to humans/arguably less of a danger to Dragaera than their fellow nobles. Thus, in Dragon, Vlad sarcastically notes the irony of calling Morrolan's army in which he is a member the "good guys", since all they are doing is trying to take some artifacts of doom/empathetic weapons so that a somewhat worse noble can't have them. Similarly, the plot of the novel, Iorich involves Vlad trying to defend Aliera after she is arrested on a charge of using illegal magic (the same type her father used and accidentally destroyed the old capitol and killed everyone there). This isn't because Aliera is innocent. Rather, it's because so many nobles break this law, that there must be a conspiracy at play for Aliera to be arrested for something she does in essentially plain sight.
Conan the Barbarian, especially Robert E. Howard's original stories. The hero is a mercenary/pirate/bandit/professional thief who isn't shy at any activity that involves cracking skulls and getting wenches and loot, but his personal code of honour means he doesn't kill wantonly and he always keeps his word. He's often pitted against fiendishly evil sorcerers, mad kings, or mad, fiendishly evil sorcerer kings. He gets a much more sympathetic portrayal when he becomes King of Aquilonia by his own hand, as he becomes a benevolent king who is defending his throne from scheming aristocrats and ambitious rivals.
J. K. Rowling was very fond indeed of doing this with her characters in the Harry Potter series. Word of God says that there were concerted efforts made to remind the readers that Harry is a flawed person (see his Order of the Phoenix "Angsting"), and is certainly no saint (his ready use of the Cruciatus curse on Amycus, and before then, Bellatrix). James (and specifically Sirius) are shown to have very good hearts overall, but could definitely be Jerkasses at times (Sirius and his treatment of Snape/Kreacher, his recklessness). Ron (who never went through what Harry did but accomplished more than most Hogwarts students could ever admit to) left Harry and Hermione in the woods. Dumbledore, of whom so many people "thought the sun shone from every orifice", made plans in his youth with another to take siege of the general Muggle population, during which time he neglected his remaining family. Paradoxically, Regulus turns out to have been not as Black as first painted - same for Snape. Draco is a tricky one, who at first doesn't turn Harry in, but later tries to capture him, accompanied by his old henchmen who, by now, are not just brainless brawns and are unafraid to kill—though given his family's precarious situation, desperation could have been a factor. Hermione enchanted the D.A. list to give scarring acne to any member who betrayed them, even if the person had limited options. She also tricked Umbridge into going into the Forbidden Forest in the middle of the night and getting attacked by centaurs.
And that's ignoring the Ministry of Magic — firmly on the gray side — which is generally darker than Harry and his friends, but still firmly better than the black of Voldemort.
Cataclysmic Horizons contains the Sodality of Gerosha, a band of superheroes many of whom are only possible because their parents were subjected to crimes against nature. They consist of a horny and insecure girl with centipede powers, an impulsive and aggressive woman with a stolen Powered Armor, a sleazy womanizer who can jump incredible heights, an angry and vindictive plant-man who is himself a victim of crimes against nature, a sociopathic Reality Warper, a Robin Hood archetype, and more. Together, they fight against Those Wacky Hebbleskins, who are essentially A Nazi by Any Other Name. Who have their own army of evil freaks, who are also only possible because their parents were victims of crimes against nature.
Martha Wells's Death of the Necromancer has NicholasValiarde, a coldblooded thief, murderer and all around Magnificent Bastard. Nic has spent years sabotaging his enemy on a Roaring Rampage of Revenge; at the start of the narrative, Nic's nearing the completion of his ultimate scheme when he and his subordinates run afoul of an unknown person using Black Magic. Somehow, this leads to the group spending the rest of the book fighting an insane mass murderer. And the reason they do it is at least partly because it's bad for business.
In Andrew Vachss's Burke books, Burke and his True Companions are mostly ex-cons who skirt or break the law frequently. They cross paths with pedophiles and other worse scum from time to time.
Near the end of Good Omens, the forces of Heaven and Hell line up across the sky, and the narrator mentions that if you looked very closely, and had been specifically trained, you could tell the difference.
Common in the works of China Miéville. Kraken, for instance, has a Lovecraftian doomsday cult as one of the nicer factions.
Tadeusz Borowski's Holocaust stories feature the occasional good character, but they don't tend to live long in the atmosphere of the camps. The characters who do survive (at least for a while) are those who're willing to steal from others, to betray each other to the guards, to help in the execution of the Jewish inmates, and to eat the corpses of their fellow prisoners so as to avoid starvation.
The Tribulation Force versus the Global Community in the Left Behind books.
In one of the books, a minor side character calls the Trib Forcers on this. Another minor character turns out to be a con man, and the Tribbers reverse the con to screw him over. And just as the main characters (and, likely, the reader) are thinking "Yeah; he finally got what was coming to him" the first minor character says something like "He's been pretending to be saved, but he's not, so he's going to Hell, forever; that wasn't enough for you so you felt the need to cheat him out of a few thousand dollars too? You're supposed to be better than he is. I don't think I can work with you guys any more."
Sisterhood Series by Fern Michaels: As the series goes on, the morality of the stories turns into this. The good guys are called the Vigilantes because they break the law in capturing a bad guy and inflicting a cruel and unusual punishment on him or her. The good guys don't kill anybody, but since their punishments tend to be of the Fate Worse than Death variety, that fact may not be very comforting. Also, the good guys have acted like big-time Jerkasses a number of times. That's okay, because the bad guys have virtually no redeeming qualities to speak of!
Percy Jackson and the Olympians: Help the Gods who are often jerkasses and sometimes cause problems, or serve a Titan who devoured his own kids and uses humanity as a source of cheap amusement or as a snack. It's Lampshaded by the main characters, other demigods, and even some of the gods themselves that while the current system is by no means perfect, it's still much better (in that the gods legitimately enjoy the modern world more or less) than what the Titans and the other primordials would want out of the world (usually starting with the destruction of human civilization as we know it).
In the Indian novel The White Tiger, some people (like Balram and Mr. Ashok) have their Pet the Dog moments, but it's pretty clear everybody else kind of sucks.
The Adversary Cycle: In The Keep, it's Rasalom vs. Nazis. Guess who's worse. Also, the Otherness vs. the Ally in general. The Ally isn't evil for the sake of being evil, but it certainly does horrible things. However, Earth needs the Ally to win because the Otherness would be much, much worse.
In Zoo City Zizi December is a cynical ex drug addict who blames herself for her brothers death. Everyone else is as bad or worse, sometimes much worse.
In Tom Kratman's Caliphate, the Imperial States of America isn't exactly a shining "white hat", but as presented in the novel it's a better option than the Caliphates (particularly the one on which the novel is focused), who are very much of the "black hat" persuasion. The other nations that get any attention aren't much better than the ISA.
In The Leeshore by Robert Reed, the Alteretics are a faction of humanity that worship an artificial god. They get "conscripts" by capturing enemy soldiers, hooking up to "glass wires" which are used to make them fanatically devoted to the priests and the artificial god. Their first act was to start bombing every manufacturing center they could find in the Solar System, and population centers as well. When they fled the Solar System towards the tiny outpost on Leeshore, the first thing they did upon landing was to kill everyone they could find, to try and prevent the inhabitants from cutting loose their Space Elevator. The Asian Alliance that is chasing the Alteretics down use the same "wires" to control the emotions of their own soldiers by reinforcing certain emotions and thought patterns, at the cost of making them extremely bloodthirsty. The two protagonists are captured by the alliance are forced into assisting them in hunting the Alteretics.
The designated heroes of The Chronicles of Magravandias are not particularly heroic, something which they and the antagonists deconstruct over the course of the trilogy. In the end, it's not that the protagonists are good so much as the villains of the story are worse.
The impetus of The Mortal Instruments series is Valentine Morgenstern's belief that all Downworlders should be exterminated, despite Shadowhunter laws that protect him. Pretty evil right? Then you learn that despite these supposed protections, prejudice against Downworlders is pretty heavily ingrained into Shadowhunter culture. The issue is lampshaded several times throughout the series, but it comes to a head at the end of City of Heavenly Fire when the Clave begins passing legislature inhibiting the freedom of faeries, despite Magnus' warnings that it will only cause future problems. If that wasn't enough, they also begin discriminating against Shadowhunters with faerie blood, abandoning one who was kidnapped by faeries and risked his life to leak vital intelligence. The author herself says that was intended to make the readers realize that the Clave isn't just flawed and in need of a few tweaks, but crippled by its own prejudice.
In the Parker novels, Parker can and will kill with dispassion, but generally prefers a minimum of violence in his heists and won't kill somebody unless they're trying to kill him directly or indirectly. He also won't take more than his share from a heist unless he's been double-crossed or otherwise screwed over.
The Daniel Faust series: Daniel's crew includes criminals, sorcerers, and a demon or two, but the monsters they go up against are even worse.
In Humane Tyranny certain members of the rebellion believe this to be a necessary evil and that in order to stop the government from killing innocent people, they must perform evil deeds that will damn their very souls. The main characters, though, wish to avoid this trope entirely and to do things the right way.
Shattered Continent: Chapter One of Caroline's Awakening features a team of mercenaries having a frank discussion about using fire or poison gas against an enemy structure that may or may not contain innocent captives. They're the protagonists. The opposing force is a bunch of half-undead, demon worshipping cultists and there are no hostages because the cultists ate them all already.
Wasp: The main character is a military secret agent working to disrupt the enemy government. Some of his actions are morally gray, such as when he booby-traps two civilian merchant ships, implicitly causing their crews' deaths. On the other hand, the enemy is a militaristic and quite oppressive empire, and their State Sec agents we meet in the book are pretty much all ruthless and brutal.
Revanche Cycle: In a story that's essentially about the price of power, even the characters with genuinely good intentions end up with blood on their hands (and several of the villains have perfectly reasonable motives for the heinous deeds they commit).
On the one hand, you have armies and gods of Eldrich Abominations who murder, torture, and worse For the Evulz. On the other, you have a madman who kills his kin, relies on his victims' souls for his vitality, and actually worships the monsters he fights. Ladies and gentlemen, The Elric Saga.
The Man in the High Castle: On one side of the world is the Empire of Japan, which remains a brutally oppressive dystopia, but they run their colonies and puppet states in a fairly negligent, liberalized way. On the other is the Greater Nazi Reich, who are borderline omnicidal maniacs driven by their war to attain racial purity by any means necessary.
In Victoria, set in a Fallout-like post-apocalyptic America, nearly all of the various successor states are some flavor of politically Ax-Crazy, ranging from human-sacrificing neo-Pagans and genocidally misandristic high-tech Amazons to cannibalistic Mexican warlords and unapologetic Nazi imitators. Victoria itself is fascist, and would be the villains in any other story, but given the crapsack nature of the setting and the author's political leanings, they're the good guys.