As many of you reported TV Tropes went offline at 2:40am EST on July 7th. It was offline for over 14 hours. The worst outage in many years. We did receive email and text alerts when it happened but unfortunately it was a major hardware failure which took quite a while to get under control.
The cause was a total failure of our database cluster. 6 of our 8 hard drives failed simultaneously resulting in a complete loss of data. We had to have our server company replace the cluster and then we had to rebuild the site from database backups. We do automatic backups every morning. Unfortunately the failure happened hours before the next backup so 24 hours of changes were lost.
To make it worse the history of wiki changes is only updated twice a week because it is over a 1TB in size. We are working on restoring that now so the history tab is blank on all pages until it's done. Editing will be offline for another 24 hours until we get that fixed. And it means we'll lose 72 hours of wiki history due to the timing of the last backup.
We will be working on optimizing our database structure so we can increase the frequency of our database backups to protect the data in the future.
We have redundant web servers on a load balancer, redundant database servers in a cluster and redundant hard drives in every server. So how did this happen? According to our server company there was a manufacturers bug in the firmware of the specific model that 6 of our 8 hard drives were on. That bug caused the disks to die after a certain number of hours running. We don't yet have all the details. They are reaching out to the manufacturer to get more information. I'll update here as I learn more.
UPDATE: (July 8th)
Editing is now enabled! History should be restored as of July 4th 10am EST. The history database imported faster than I expected (4 hours to decompress 1.1TB sql file, 12.5 hours to import)
The only thing I haven't done yet is purge the CDN cache. You must logout to view a page cache. Logged-in users get the live site. Not all pages are still cached and they do expire.
I'll hold off on purging the cache for a few more hours. If there is some specific edit you remember doing during those 24 hours that were lost you may be able to find it by logging out and viewing the cached page. Then login and make that edit again.
Edited by itcdr on Jul 8th 2020 at 7:12:45 AM
The sextuple cock-up... Now that's a name.... 
Glad the site is back! It makes our daily routine better :)))
- Fly, robin, fly! - ...I'm trying!
Phew! It could have been way worse! Thanks to the guys at TVT for fixing this!
So where are we going from here?
Big Catch, Sparkle Edition
I hope we can figure out where the videos went. I uploaded two of them Monday night before the server crash; they both went through. Now I'm stuck with the 10% glitch again and it sucks because the videos were properly uploaded and now they're refusing to upload again. If there's any chance of getting the information from Vimeo and restoring them, that'd be awesome.
Current Project: Incorruptible Pure Pureness
Also, what was with the Wiki briefly coming back blank?
Is the save profile save button not supposed to be there?
Jawbreakers on sale for 99¢
Oh yeah, I saw the temporary restoration. Nothing existed. No articles, or forum edits, or even accounts. It was so weird...
Well everyone, LEGO Rewind is finally back on TV Tropes! Again, this couldn't have been accomplished without Albert 3105's discovery of the cache exploit and screen-capping, so special thanks. Any and all Wiki Magic and crosswicking is more than welcome.
An open mind and compassionate heart are among the most important qualities we can have.
I had freaked out a bit. Thought I was hacked. Then I tried to go to this site on my phone which I had never logged on with. Said it couldn't reach the server. I instantly felt relief but I wondered what happened. Glad to see it's over now.
FF.net, AO 3, and Quotev as Archdemon Slayer
Creator of Universes
I honestly freaked a bit when this happened. I thought there was something wrong with my connection or my computer/phone. I was lucky to only loose a minute's worth of editing, but my heart goes out to everyone who lost hours' worth of research and editing to what I'm going to call The Day the Tropes Died.
I am the captain of my soul. - Invictus
Logic Girl
That's a bit of a morbid name. I was thinking some names for the event could be:
- The 2020 Glitch
- The Not-So-Great Crash
- The Hard Drive Incident
groovin'
![[up]](https://static.tvtropes.org/pmwiki/pub/smiles/arrow_up.png "[up]"){kind=icon}
- The Sextuple Cock-Up
- Tropes Machine Broke
- The Fairly Bad Crash Of 2020
- Withdrawal Day
- C-C-C-C-C-Combo Crash
Edited by JXZ on Jul 8th 2020 at 1:54:16 PM
my brain is a computer with 4k of ram. this is a jokes wiki
Big Catch, Sparkle Edition
"Those Two Heroic Hard Drives" Day
Current Project: Incorruptible Pure Pureness
Veteran Editor IV
I prefer "The Unlucky 7/7 Day", to be honest, or something similar (some may suggest something based on Persona 5 as it's the in-game day Atlus initially blocked streaming from due to spoiler stuff, but IDK).
Edited by Piterpicher on Jul 8th 2020 at 7:57:56 PM
Currently mostly inactive. An incremental game I tested: https://galaxy.click/play/176 (Gods of Incremental)
Angry babby
"The 2020 Glitch" 2020 is a glitch.
I do some cleanup and then I enjoy shows you probably think are cringe.
One of the strangest human beings you will ever
Saving the articles will restore them. I tried with Magia Record’s Nightmare Fuel and Tear Jerker pages (which got wiped completely) and saving them restored the pages. I’d recommend hunting down the pages that were wiped and saving them to restore them.
Just a guy on TV Tropes who loves Anime and Video Games. Savy?
We should really look into archiving this site somewhere, to ensure there's no chance of the website vanishing without a backup/
Friends forevermore
If we absolutely have to give this a name, "The Crash of 2020" probably works best. It's not a Great Crash.
I had a dog-themed avatar before it was cool.
Was this due to some Demand Overload?
Nope, six of TVTropes' eight hard drives all just fucked off into the sunset simultaneously. pardon the language
Anyway, as a name for this endeavor, I like "The Unlucky 7/7 Day" and "C-C-C-C-C-Combo Crash".
Jawbreakers on sale for 99¢
Oh, I understand. I just got confused because the same day, a BTS album release crashed Twitter (no wonder why they have their own entry on Demand Overload), but it turns out it was hardware, like what commonly happens to the Internet Archive. Luckily, the only edit I made was to Screwed by the Network concerning how the COVID-19 pandemic affected Broadway. Also, when I re-added that, I added an example about a similar incident (the revival of The Rocky Horror Show ending due to declining tourism after September 11th.)
Unofficial Wiki Curator for Complete Monster
Um, shit. I'm trying to add something to Sherlock Holmes In The Twenty Second Century S 1 E 06 The Adventure Of The Deranged Detective, and it won't let me.
CM Dates; CM Pending; CM Drafts
I'm helping!
That's because the page title is 77 characters long. I forget the limit, but it's definitely less than that.
Trouble Cube continues to be a general-purpose forum for those who desire such a thing.
Veteran Editor IV
Again, too long title, not the crash. It's 77 chars, the limit's 64. I recommend removing the episode title.
Currently mostly inactive. An incremental game I tested: https://galaxy.click/play/176 (Gods of Incremental)
Thank God! I've missed you, guys!![[tup]](https://static.tvtropes.org/pmwiki/pub/images/Thumbs_up_emoticon_3268.png "[thumb up]")
for the hardworking people who fixed it.
writer/producer/director/visionary/troper Ai-Fan The Master Of The Romantic (But Deadly) Drinking Game And Haiku™