As many of you reported TV Tropes went offline at 2:40am EST on July 7th. It was offline for over 14 hours. The worst outage in many years. We did receive email and text alerts when it happened but unfortunately it was a major hardware failure which took quite a while to get under control.
The cause was a total failure of our database cluster. 6 of our 8 hard drives failed simultaneously resulting in a complete loss of data. We had to have our server company replace the cluster and then we had to rebuild the site from database backups. We do automatic backups every morning. Unfortunately the failure happened hours before the next backup so 24 hours of changes were lost.
To make it worse the history of wiki changes is only updated twice a week because it is over a 1TB in size. We are working on restoring that now so the history tab is blank on all pages until it's done. Editing will be offline for another 24 hours until we get that fixed. And it means we'll lose 72 hours of wiki history due to the timing of the last backup.
We will be working on optimizing our database structure so we can increase the frequency of our database backups to protect the data in the future.
We have redundant web servers on a load balancer, redundant database servers in a cluster and redundant hard drives in every server. So how did this happen? According to our server company there was a manufacturers bug in the firmware of the specific model that 6 of our 8 hard drives were on. That bug caused the disks to die after a certain number of hours running. We don't yet have all the details. They are reaching out to the manufacturer to get more information. I'll update here as I learn more.
UPDATE: (July 8th)
Editing is now enabled! History should be restored as of July 4th 10am EST. The history database imported faster than I expected (4 hours to decompress 1.1TB sql file, 12.5 hours to import)
The only thing I haven't done yet is purge the CDN cache. You must logout to view a page cache. Logged-in users get the live site. Not all pages are still cached and they do expire.
I'll hold off on purging the cache for a few more hours. If there is some specific edit you remember doing during those 24 hours that were lost you may be able to find it by logging out and viewing the cached page. Then login and make that edit again.
Edited by itcdr on Jul 8th 2020 at 7:12:45 AM
The Dishonorable
Christ. 6 out of 8 hard drives failed.
If that's not an argument in favor of permanent and mandatory transition to solid state drives or solid state hybrids, I don't know what is.
"The devil's got all the good gear. What's God got? The Inspiral Carpets and nuns. Fuck that." - Liam Gallagher
Lost in Space
That's the thing. We were supposed to have backups, and we were supposed to have redundancy in case something like this happened. Nobody could have anticipated a firmware bug that caused six hard drives to fail at the exact same moment. This goes beyond "freak accident" to "act of malicious deity".
Anyway, backing up the HTML is worthless, unless you want to use it to manually re-enter pages that had data lost. The underlying data itself isn't really scrapeable, and we'd block anyone trying.
Edited by Fighteer on Jul 7th 2020 at 7:35:46 AM
"It's Occam's Shuriken! If the answer is elusive, never rule out ninjas!"
Well, it is 2020. Incredible catastrophes is what this year is doing best.
Disasturbator
Big Catch, Sparkle Edition
I wonder what would've happened if all 8 hard-drives failed? Pretty weird.
2020 sure is a strange one.
Current Project: Incorruptible Pure Pureness
The Dishonorable
Edited by Mario1995 on Jul 7th 2020 at 7:41:11 AM
"The devil's got all the good gear. What's God got? The Inspiral Carpets and nuns. Fuck that." - Liam Gallagher
Disasturbator
Do not entertain the possibility of all eight failing!
New theme music also a box
indighost | he/them
I'm glad we didn't lose everything! 
Semain Calm
alright though; let's throw a party for the 2 that survived this experience
and the public won't dwell on my transmission cause it wasn't televised.
Big Catch, Sparkle Edition
![[up]](https://static.tvtropes.org/pmwiki/pub/smiles/arrow_up.png "[up]"){kind=icon}
-Blows party horn-
Predator Loyal to the Zerg Swarm
Glad we didn't lose everything. This site has a special place in my heart.
If we had a real life section for Wham Episode, I'd post this day in that section. Maybe post it in Web Originals?
Lost in Space
The drives are in a RAID array that would have let us keep running if up to four had failed. Anything more than that and we were toast.
"It's Occam's Shuriken! If the answer is elusive, never rule out ninjas!"
uwu
I initially thought the server crash was a fault of my computer, but it wasn't the case. Glad to see that we haven't lost much info before the crash. This wasn't exactly the birthday present I was expecting, but you'll never know what'll happen in 2020.
THE GOLDEN AGE WILL RETURN AGAIN!
indighost | he/them
(confetti shoots out from the air cannons)
Edited by AngrokVa on Jul 7th 2020 at 7:44:49 AM
Xbox/PlayStation: IndiGhost77 | on semi-hiatus
I admit, first thing this morning seeing notice that we had catastrophic hard drive failure - nothing like that to wake you up.
Actually, that's a fair question: are we getting a page for this like we have one for the Great Crash?
Musician/Composer
Well, I’m glad that most of the situation has been resolved. Keep up the good work.
Edited by gjjones on Jul 7th 2020 at 7:49:22 AM
He/His/Him. No matter who you are, always Be Yourself.
Semain Calm
https://tvtropes.org/pmwiki/pmwiki.php/Timeline/TVTropes
someone should put this on the timeline once editing is back up
and the public won't dwell on my transmission cause it wasn't televised.
Angry babby
There's something funny about the "Mario Pissing" draft on the timeline being immediately followed on the timeline by the worst server crash we've had in years.
Edited by mightymewtron on Jul 7th 2020 at 7:49:34 AM
I do some cleanup and then I enjoy shows you probably think are cringe.
#BoomSilverTheVampire4Lyfe
Great, now I have to rewrite portions of the article TerraGenesis from memory.
Content Warning: My posts may involve my actions dealing with R-rated or Not Safe for Work content. Same for my edit history.
Thanks for all the encouraging words. It's been a very long scary day. The computer screen is getting blurry.
Pulling down the edit history from the backup server is almost done. Once it finishes decompressing I'll start the import to the database which will probably take 20 hours. I'll continue monitoring the progress tomorrow and update this thread when we'll be able to re-enable editing.
Edited by itcdr on Jul 7th 2020 at 4:56:01 AM
The Tropes Are With Me.
A whole day without TV Tropes...I don't know how the hell I got through this...
Hmm. 6 of 8?!? Simultaneously?!? Ye vish! That would kill even full mirroring (i.e., RAID 1).
Oh Captain!~
Well, thank the universe that just the stuff from the past day got lost. Reposting in threads feels like deja vu.
Hey how you doing well I'm doing just fine I lied I'm dying inside
The only thing I really remember doing was uploading two videos I still...ya know...have. So when we're ready to edit again I can just re-add those.
Current Project: Incorruptible Pure Pureness