A Series of Unfortunate Events: Count Olaf got less and less threatening as the books went on, although to some degree other villains picked up the slack.
Visser Three from Animorphs. He's completely immoral and monstrous, but as a consequence of appearing in almost every book and not killing the Animorphs, he quickly starts to come across as a blundering clod. Visser One even acknowledges this in The Visser Chronicles, comparing him less-than-favorably to the Helmacrons. While he does get promoted to Visser One towards the end, he still doesn't undo a lot of this.
It becomes funny when even Visser Three starts noticing the effect. "Would it be too much to ask for one of you to actually HIT SOMETHING?!?!"
D.Metria the Demoness from the Xanth series, started off as a fairly malevolent seductress, but with each subsequent appearance became less threatening, to the point that by the time she was "replaced" by her insane doppleganger, D.Mentia, she was Xanth's version of Mr. Mxyzptlk (The Superfriends version, at that). In more recent books this is justified by her having acquired half of a human soul, which gives her a conscience. She can still be mischievous, but is no longer malevolent.
In Tales of MU, Puddy and Sooni start out as the Manipulative Bastard and the Alpha Bitch, respectively, but are eventually reduced to being pathetic losers who struggle to keep even a couple people under their control. The worst aspects of the transition are probably a result of Webcomic Time: the change takes several months of real world writing time, but just a couple weeks story time.
An inversion of this trope occurs when Severus Snape and Draco Malfoy are presented as mere Jerkasses who like giving Harry and company hell for the first five books of the series, but at the beginning of book six, both are presented as high ranking members of Voldemort's army, the Death Eaters. It does turn out however that Snape was a Double Agent for the Order of Phoenix, and Malfoy was incapable of coping with actually being evil.
Draco's example is a bit more complex. From the start, he was Harry's greatest rival in school, and the source of most of Harry's non-Voldemort-related troubles. He's even called Harry's arch-nemesis in Book 2. However, once Harry starts facing down Voldemort more, he starts to see Draco less as a menace and more of a nuisance. He basically laughs in Draco's face when the latter threatens him at the end of Book 5. So the trope gets played straight, and then get inverted as mentioned above.
Lucius Malfoy. He is introduced as a sinister and cunning master-manipulator, who, while maintaining a benevolent and charitable public image, actively and ruthlessly pursued his ambitious goals, descending to threatening whole families and unleashing an ancient monster on a school. Next time he's just The Dragon, and after his failure and consequent fall-from-grace, he's reduced to an unnerved bystander with little to no involvement in the action.
The Wheel of Time series features significant villain decay with regards to the Forsaken, the 13 most devoted human servants of the Dark One. Initially presented as uber-badasses from the Age of Legends wielding powers most modern people could not begin to comprehend and being trained as scientists, generals and geneticists, the Forsaken get defeated repeatedly by the present-day heroes. Partly, this is because the Forsaken's reputation got exaggerated during the 3000 years the spent imprisoned, partly it's because they lack the support network they had in their prime, but whatever the reason, there's still a big gap between their myth and the reality, which was one of Jordan's more anvilicious points in his series (the gap between hearsay and reality, specifically).
The Young Wizards series averts this rather neatly, because a), the Lone Power has been defeated without great sacrifice a grand total of...once (and in a subsequent encounter, another aspect commented that that version of Itself was just plain stupid), and b) because It exists outside of time, dipping into our universe wherever and whenever It pleases, It can be decisively defeated in one place and simultaneously be an active threat elsewhere.
Sang-Drax from The Death Gate Cycle series was introduced in the fifth book as a Magnificent Bastard manifestation of the Big Bad that could play Haplo like a fiddle. While he's still cunning in the next two books, he gets a whole lot sloppier, downgrading him to a literal Smug Snake. He finally dies when a room caves in on him. This isn't as lame as it sounds because said room was filled with magic that was antithetical to him, but still — he really should have seen it coming.
It becomes apparent towards the end of Peter Pays Tribute that the Gray God needs Peter, his acolyte, more than Peter needs him. Also, Briskle is more crazy than competent.
The Lord of the Rings: Tolkien often does this deliberately, but still puts the less-powerful villains in situations where they can get the upper hand. Saruman goes from needing a massive army, a wizard, and more to stop him, to being somebody who could be defeated by a mob of angry Hobbits. Of course, by then Gandalf has cast him from the order of Wizards and broken his staff. Gollum is another example - he finds the One Ring to Rule Them All, and first uses it for murder and theft, but eventually crawls into a cave and uses the Ring's power to catch fish. The Ring doesn't particularly care for this. In fact, this is one of the core themes of the stories, because Evil Is Petty it eventually loses everything that once made it great and noble.
The Silmarillion: This is explicitly canon for Melkor/Morgoth. He starts out out-powering everything else in the universe except for God and being quite cunning to boot, but as the book progresses he is drastically weakened after squandering his power on evil creations and getting Shapeshifter Mode Locked, and his cunning goes down the drain as he goes increasingly Ax-Crazy and ends up as Orcus on His Throne. He ends up being permanently wounded in a fight with the Elf Fingolfin, despite killing Fingolfin, then has his face scarred by an eagle.
That said he still proves to be a really terrible villain who is responsible for much of the troubles of the First Age while in his Fortress. Eventually it takes the Valar to defeat him.
Sauron begins as one of the most powerful Maia but puts most of their power in their Ring of Power. And when he is caught in the destruction of Numenor he is left unable to take a fair form. That said, despite not having the ring Sauron still comes very close to taking over Middle-Earth and would have succeeded if not for Gollum's unlucky slip.
One consistent theme is that while the power of evil drops, the actual threat remains constant as far as Middle Earth is concerned. The Valar, Mair, and Elves retreat across the sea over time, leaving only the few free Men and the Hobbits to make the last stand with only a single Wizard supporting them.
In The Legend of Drizzt there's Artemis Entreri after the first few encounters with him, as Drizzt no longer wishes to fight him, and at one point refuses to kill him despite the perfect opportunity. Also, Entreri is getting old, whilst Drizzt is still in his prime.
The Heralds of Valdemar series plays this trope intentionally with the overarching Big Bad, Ma'ar. He starts out in the Mage Wars prequels as a frighteningly powerful, ruthless Well-Intentioned Extremist who can rival Great Mage Urtho in sheer power. Even worse, he conceives of an amazingly effective My Death Is Just the Beginning gambit involving hiding his soul in a pocket of the nether plane until a blood descendant learns to wield magic, at which point he steals the body, destroying its original soul in the process, and embarks on a new plan to Take Over the World. As he is constantly thwarted over the centuries, however, his spirit becomes increasingly petty and narcissistic, and eventually he grows careless enough to sow the seeds of his defeat when he fails to destroy the soul of his latest possessee, An'desha. Also a case of divine intervention, as it turns out that the Gods were tacitly abetting his scheme because they needed his knowledge to avert a repeat of the Cataclysm 3000 years later.
Deliberately invoked and deconstructed with Cersei from A Song of Ice and Fire. She started out as the puppet master behind King Robert and became one of the most feared characters in the series when she declared herself Queen Regent after his death... but promptly ran herself straight into the ground the second the checks on her power were removed. As the power went to her head, her schemes became less competent and more deranged over time, and while she was still somewhat feared it was more because she was psychotically unstable and overly trigger happy.
This becomes even more obvious when Petyr "Littlefinger" Baelish, a villain who was previously viewed as harmless (and still is considered as such in-universe) before throwing that façade out the windowalong with his wife, finds Cersei's stupidity and the fact that she still believes herself to be a Magnificent Bitch rather amusing
Matthew Luzon in the Petaybee book, Power Play. Though he was supposed to do a live-to-fight-another-day sort of thing, he ended up just hiding and sending other people to do his dirty work. Intergal counts as well, especially in the Twins Of part of the series. Their attempts to reclaim Petaybee become less about Petaybee and more about payback for losing it in the first place.
Happens to Lord Ombra in the series Peter and the Starcatchers. In Peter And The Shadow Thieves, all Ombra has to do is have his shadow overlap with yours, and immediately he steals your shadow, giving him full access to all your memories and turning you into an utterly obedient slave with no further effort or maintenance on his part required. In "Peter And The Secret of Rundoon" onward, he... can't. Being as Ombra in his first incarnation was ludicrously overpowered, it was a choice between Villain Decay or Only the Author Can Save Them Now.
In Death: This is mostly avoided by having a new murderer in each book. This still happened with Isaac McQueen in New York To Dallas. He started out as a cunning pedophile who had never been caught and he seemed to avoid even being noticed in the first place... until Eve took him down as a rookie. She wasn't even out to arrest him, she was just questioning him on a matter that was not really related to him, and he attacked her when she wouldn't leave. Twelve years later, he escapes from prison trying to get Revenge on Eve and still seems untouchable. However, by the end of the story, he turns out to be a pedophile who has lost a lot of his intelligence, and his ability to make even basic decisions. Dr. Mira even explains that the 12 years in prison, having the power to make decisions taken away from him in that time, and breaking most, if not all, of his patterns in illogical ways have devolved him!
La Quête d'Ewilan: When the Ts'liches were first introduced, they were described as the "ultimate predators" that only one man ( Edwin) was able to defeat in a fight. As the story goes, Edwin goes from being able to defeat one of two of them in a fight, to defeat six of them at the same time. After that, their last appearance ends with the last of them being exterminated with ridiculous ease, with almost all character getting to kill one.
Hannibal Lecter: Hannibal "The Cannibal" fell hard. The first two books, Red Dragon and The Silence of the Lambs, established Lecter as a highly intelligent and manipulative psychopath with little in the way of redeeming characteristics. In Dragon his only major contribution to the storyline is encouraging the villain to slaughter the hero's wife and son. Lambs gives him a small measure of likeability due to his actions leading to the triumph of the hero, the downfall of the villain and the rescue of the victim but his exact motivations for doing so are up to interpretation (and he also kills five innocent people who were in his way). Thanks to Anthony Hopkins mixing in a large dose of Affably Evil with the already potent levels of Evil Is Cool the character was upgraded to a something of an anti-hero in Hannibal. With the addition of a Freudian Excuse and a Robin Hood-esque preference to only killing assholes he was mostly defanged and was the star of his own story, Hannibal Rising.
However the Makuta species in general and their Mooks the Rahkshi did suffer from this a bit. The Makuta who got introduced later acted way more down-to-earth than Teridax and displayed comical traits, what more, one of them, the similarly named Tridax got killed under mere seconds by one good guy. The Rahkshi also went from being near-unstoppable demons to simple foot-soldiers that fell to swords and some fire.
In Vampire Academy, Victor Dashkov goes from the lethal Big Bad of the first book to somewhat of a lackey for others later on.
Section 31 in the Star Trek Novel Verse has been getting hit hard with this lately. In their original television appearances, Section 31 agents could dance circles around other foreign intelligence services and had some truly devious, if extremely morally questionable, Batman Gambits going on. Now, their operations are routinely foiled by Julian Bashir and his allies. In fact, despite Bashir ruining several of their master plans, Section 31 remains absolutely obsessed with recruiting him to the extent that it borders on Stalker with a Crush territory. They're portrayed as so incompetent at their mission of protecting the Federation that they fail to prevent the assassination of President Nanietta Bacco but also don't know that her successor is a Cardassian mole. It seems David Mack realized this and Star Trek Section 31 Control undid this, hard. All those failures? The AI controlling Section 31 planned for them and recruited Bashir to help take down the agency.
Inverted in The Belgariad. Belgarath talks smack about Ctuchik throughout the series, obviously considering him a two-penny warlock with delusions of adequacy. However, when the two face off, Ctuchik manages to fight Belgarath (who is at that point seven thousand years old, widely considered to be the most powerful sorcerer in the word) to a standstill.
For a slightly less deadly kind of enemy, by the last Bridget Jones book, the arch cad and menace to happy relationships Daniel Cleaver has been demoted to eccentric friend who babysit's Bridget's children.
In the final book of The Dark Tower series, Walter o'Dim/Randall Flagg and the Crimson King, the two main villains up until that point, are reduced to almost non-entities. In particular Flagg, who had once been considered King's 'ubervillain', is easily dispatched in one chapter by a newly introduced character, who finds him pathetic. This is especially troublesome to some fans, since Flagg has been especially troublesome to numerous Stephen King characters over the years, even surviving (or, more accurately, respawning after) a nuclear blast. Having Mordred kill him is obviously to establish Mordred as even more sinister and dangerous than Flagg, but that's a bit hard, if not impossible, after developing the Flagg character for decades and across numerous books. Made worse by the fact that Mordred's death is very anticlimactic (weakened by food poisoning due to eating the disease-ridden corpse of Lippy the horse, Roland kills him with one shot).
The Crimson King goes from a shadowy menace who's behind everything happening, who's terrifying because he's unknown and unseen, to a cackling madman throwing grenades from a balcony. He's effectively destroyed by a pencil eraser.