As many of you reported TV Tropes went offline at 2:40am EST on July 7th. It was offline for over 14 hours. The worst outage in many years. We did receive email and text alerts when it happened but unfortunately it was a major hardware failure which took quite a while to get under control.
The cause was a total failure of our database cluster. 6 of our 8 hard drives failed simultaneously resulting in a complete loss of data. We had to have our server company replace the cluster and then we had to rebuild the site from database backups. We do automatic backups every morning. Unfortunately the failure happened hours before the next backup so 24 hours of changes were lost.
To make it worse the history of wiki changes is only updated twice a week because it is over a 1TB in size. We are working on restoring that now so the history tab is blank on all pages until it's done. Editing will be offline for another 24 hours until we get that fixed. And it means we'll lose 72 hours of wiki history due to the timing of the last backup.
We will be working on optimizing our database structure so we can increase the frequency of our database backups to protect the data in the future.
We have redundant web servers on a load balancer, redundant database servers in a cluster and redundant hard drives in every server. So how did this happen? According to our server company there was a manufacturers bug in the firmware of the specific model that 6 of our 8 hard drives were on. That bug caused the disks to die after a certain number of hours running. We don't yet have all the details. They are reaching out to the manufacturer to get more information. I'll update here as I learn more.
UPDATE: (July 8th)
Editing is now enabled! History should be restored as of July 4th 10am EST. The history database imported faster than I expected (4 hours to decompress 1.1TB sql file, 12.5 hours to import)
The only thing I haven't done yet is purge the CDN cache. You must logout to view a page cache. Logged-in users get the live site. Not all pages are still cached and they do expire.
I'll hold off on purging the cache for a few more hours. If there is some specific edit you remember doing during those 24 hours that were lost you may be able to find it by logging out and viewing the cached page. Then login and make that edit again.
Edited by itcdr on Jul 8th 2020 at 7:12:45 AM
Might be minor but... the "Since January 1, 2012 this article has brought X people to the wiki from non-search engine links." counters from the Related Pages were rolled back as well?
With Great Power, Comes Great Motivation
So apparently, this crash was caused by faulty hardware. How are measures going to be taken to prevent this from happening again.
Also, lol I only now found out about the Mario Pissing Launch Pad, and threw a bomb in there for shits and giggles.
Well, I must be the unluckiest schmuck on the website, because it just so happens that I launched a work page on the very night before the crash! It was for a YouTube series called LEGO Rewind (playlist here for those interested
). Please let me know if there's any way to restore the article and its YMMV and Trivia sub-pages.
![[up]](https://static.tvtropes.org/pmwiki/pub/smiles/arrow_up.png "[up]"){kind=icon}
I managed to access a cached version of your pages and screenshoted the entire thing. Will upload to imgur ASAP.
EDIT: Here it is: https://imgur.com/a/ygfmadc
Edited by Albert3105 on Jul 7th 2020 at 1:22:13 PM
The Cosmopolitan Fictioneer
Man, the way the hardware failure was described reminded me of the tale of SCUTTLE from the SCP Foundation site.
Veteran Editor IV
Thanks for bringing the wiki back so quickly despite the insurmountable odds. Admittedly, I could have helped with the crash since I always wanted to make a thread about removing wiki history, to make the load easier since those pages take a crapton of space (I know the problem is a hard drive bug, but those 1TB backups are a pain), but damn was I lazy. I know you'll want to make backups more frequently, but I think I have an idea that'll make the backups lighter and history more useful, though I'd only need moderator intervention I think. I'll start it later tomorrow probably, since now I do have a reason for it and I want to wake until editing's back. And this whole event is going on Timeline.TV Tropes.
{edited since I'd rather wait until this siituation cools out, maybe the admins will come up with some other solution, IDK).
Edited by Piterpicher on Jul 7th 2020 at 8:00:48 PM
Currently mostly inactive. An incremental game I tested: https://galaxy.click/play/176 (Gods of Incremental)
groovin'
wouldn't have guessed that tv tropes would fit on only 8 hard drives. If my understanding of RAID is correct, then it would technically fit on just 4 hard drives too. just how big are these drives?
my brain is a computer with 4k of ram. this is a jokes wiki
Good to see the site back again.
I can't be 100% sure but it seems like I have been pretty lucky and the only edits I lost were ones to other edits that were also lost (along with some forum posts).
Knowledge is Power, Guard it Well
Veteran Editor IV
The largest disks that are widely available are 16TB HDD disks and 4TB SSD disks. To be honest, the wiki isn't the most massive (at least, it'd pale in comparison to something like Youtube) due to consisting mainly of text (the images can't be larger than 2MB). The histories are pretty damn heavy, but again, I'll make a new thread tomorrow, once people cool out.
Edited by Piterpicher on Jul 7th 2020 at 7:59:54 PM
Currently mostly inactive. An incremental game I tested: https://galaxy.click/play/176 (Gods of Incremental)
Purposefully Untitled
So, despite the whole "absent edit history" thing, both the Following and X Troper's Edits functions work just fine.
EDIT: The material from the last 24 hours is still gone, though.
Edited by Lermis on Jul 7th 2020 at 9:02:15 PM
SpaceBattles.com fanworks (unnoficial) index in my Sandbox.
insert commentary here
Dang, I'm not even sure if I can remember all the edits that were lost from Monday.
At least I didn't make many forum posts so I'm not torn up about the loss of those.
Thanks for getting the site up and running again so quickly!
The edit histories for all the pages are empty. Can those be restored or are they gone forever?
Veteran Editor IV
Yes, they will be restored eventually, I think the pinned post says they're trying to bring them back as well.
Edited by Piterpicher on Jul 7th 2020 at 8:13:47 PM
Currently mostly inactive. An incremental game I tested: https://galaxy.click/play/176 (Gods of Incremental)
Angry babby
Because editing is locked, I'm using the To-Do Lists for the first time in my life.
Edited by mightymewtron on Jul 7th 2020 at 2:24:44 PM
I do some cleanup and then I enjoy shows you probably think are cringe.
@Nouct: Maybe you can look at your browsing history to help figure it out?
Likes video games
I'm glad to see this website is online again.
The four eyes of Isabelle
TV Tropes is back up? Time to waste my time again. 
rollin' on dubs
YAY! We don't have to face reality anymore!
2020, what are you doing! STHAP!
All night at the computer, cuz people ain't that great. I keep to myself so I won't be a case on The First 48
Secret Agent2583
I'm glad it's back, I was worried sick!
Secret Agent2583
Also, all the other videos that I think were uploaded on Sunday / Monday aren't back yet.
Yesterday showed me just how dependent I am on this site. Can't wait for editing to be back.
...
I've found that at least one of my recent edits is gone. Will it be restored from the edit history that was saved or will I have to re-write it when editing is back?
According to the main post, edit history was backed up less frequently than the actual content, meaning three days of edit history was lost as opposed to the day of actual edits that were lost. Unfortunately, you'll probably have to recreate that edit, unless that can be retrieved from an external cache.
Lost in Space
We are waiting to find out the results of the restoration of the history tables. That's the big one: over 1 TB of data. It's by far the largest single component of the site's database. We'll be looking into ways to optimize it going forward.
Gotta give the admins some time to rest; they were up very early yesterday and worked for well over 16 hours.
"It's Occam's Shuriken! If the answer is elusive, never rule out ninjas!"
Kudos to the admins for getting the site up and running again. The loss of 72 hours of edits is bad but I still remember The Great Crash of '08, and it was much worse back then.