As many of you reported TV Tropes went offline at 2:40am EST on July 7th. It was offline for over 14 hours. The worst outage in many years. We did receive email and text alerts when it happened but unfortunately it was a major hardware failure which took quite a while to get under control.
The cause was a total failure of our database cluster. 6 of our 8 hard drives failed simultaneously resulting in a complete loss of data. We had to have our server company replace the cluster and then we had to rebuild the site from database backups. We do automatic backups every morning. Unfortunately the failure happened hours before the next backup so 24 hours of changes were lost.
To make it worse the history of wiki changes is only updated twice a week because it is over a 1TB in size. We are working on restoring that now so the history tab is blank on all pages until it's done. Editing will be offline for another 24 hours until we get that fixed. And it means we'll lose 72 hours of wiki history due to the timing of the last backup.
We will be working on optimizing our database structure so we can increase the frequency of our database backups to protect the data in the future.
We have redundant web servers on a load balancer, redundant database servers in a cluster and redundant hard drives in every server. So how did this happen? According to our server company there was a manufacturers bug in the firmware of the specific model that 6 of our 8 hard drives were on. That bug caused the disks to die after a certain number of hours running. We don't yet have all the details. They are reaching out to the manufacturer to get more information. I'll update here as I learn more.
UPDATE: (July 8th)
Editing is now enabled! History should be restored as of July 4th 10am EST. The history database imported faster than I expected (4 hours to decompress 1.1TB sql file, 12.5 hours to import)
The only thing I haven't done yet is purge the CDN cache. You must logout to view a page cache. Logged-in users get the live site. Not all pages are still cached and they do expire.
I'll hold off on purging the cache for a few more hours. If there is some specific edit you remember doing during those 24 hours that were lost you may be able to find it by logging out and viewing the cached page. Then login and make that edit again.
Edited by itcdr on Jul 8th 2020 at 7:12:45 AM
Semain Calm
here robert if you want that file back
I think I tried that method before a few years ago, but thanks
I had wondered why the site was down all day. Glad to know it's been saved somewhat. Yeah, I know the feeling of how bad a hard drive dying on you can be. I had one of my personal hard drives just flat-out die on me without warning, and it had all of my important photos, songs, pictures, stories, and other things on it. I had to send it to a computer company to get the data extracted and waited a month to get it back on a new hard drive.
Well, this was totally unexpected. Hopefully things return more or less back to normal very soon.
A procrastination in of itself
Well that was rather 2020 of the server. Glad to know we’re getting almost everything back.
“And the Bunny nails it!” ~ Gabrael “If the UN can get through a day without everyone strangling everyone else so can we.” ~ Cyran
Face me now, Bitch!
Man, I don't think I've seen an outage for this site that was this bad before.
Glad things are better.
Edited by Demongodofchaos2 on Jul 7th 2020 at 9:22:27 AM
Watch Symphogear
It was weird to not be able to explore this site and check on my watchlist for a few hours; I tab back in regularly. Glad you guys got things sorted and are working on getting TV Tropes back to 100%. Good luck, and may the recovery be speedy.
Edited by SonicWind on Jul 7th 2020 at 9:29:27 AM
Well, that was close. I recently finished writing a short story and planned to give it a tvtropes page; instead I decided to give it one final update first and delay the page until the update was done. If I did put it on tvtropes, 50+ page changes (the story's page and all the trope references) would have been zapped into oblivion by the crash.
And don't worry, the list of tropes is safe on my PC.
(Copied and pasted from another thread)
Just wondering, are the wipe-outs randomized?
The page I created yesterday afternoon, is gone (f.....), so I went back to check a few other pages I've made recently.
A page I made 3 days ago is still there, but another unrelated page from 4 days (I think? I launched that at night, I think it was at least 4 days), BOOM, gone too. Are they being deleted randomly, Infinity Gauntlet-style?
Angry babby
![[up]](https://static.tvtropes.org/pmwiki/pub/smiles/arrow_up.png "[up]"){kind=icon}
Only pages made in the past 24 hours seem to have been wiped, but the edit history from the past few days is gone.
Edit: Wait I didn't see that another page from longer ago got wiped. If it's a recently launched trope from TLP, it might have gotten unlaunched if people deemed it rogue?
Edited by mightymewtron on Jul 7th 2020 at 9:31:07 AM
I do some cleanup and then I enjoy shows you probably think are cringe.
Gangsta Green
Whoa. I didn't realize how serious this was.
Jermaine Deon Carter
Wow! So that was why the website was restarted to yesterday morning. I lost my recent editing on the Punch-Out!! page, which means I have to wait another day before I can edit again. Fortunately, I had to add Charlie Daniels' death again on the On-Topic forums, since it was wiped out in the crash.
I'm glad you guys were able to at least save most of the information on the site after the major crash happened. Thank you for your hard work.
Mightymewtron, it's not just recent pages; my first ever edit (Anachronism Stew) was 8+ years ago, yet the history was zapped.
Edited by DragonAtma on Jul 7th 2020 at 9:36:13 AM
They're working on re-adding the page histories.
Speaking of, what are we gonna do about the discovery that the histories of at least some pages can be viewed while logged out?
The importation of our history from back-ups is just getting underway. Once caught up, we should have edit histories up to 6AM PST yesterday.
Lost in Space
Note that our distributed caching system may preserve copies of article history from the past 24 hours for users who are logged out, but only until the cache updates.
Edited by Fighteer on Jul 7th 2020 at 9:39:47 AM
"It's Occam's Shuriken! If the answer is elusive, never rule out ninjas!"
Can we save that before it goes?
Redmess
Looks like we need a more robust server setup.
Maybe distribute discs across vendors so that this kind of failure doesn't take out the entire server storage.
Optimism is a duty.
I was worried about what was going on. Good job getting the site up and running again with only minimal damage.
Also good to know that we didn't get something identical to The Great Crash.
Again, I have to ask: what are we doing about the edits that can be seen while logged out?
You could put together a clean-up thread to restore as many as you can. As Fighteer said, you're working against a cache.
Can the cache be delayed? When does it update?
Probably not, and I don't know, sorry.
Guess I shouldn't have deleted that Word file for the movie whose page I launched yesterday afternoon...
(edit: went back to check it again, yep it's lost) well it's only one page, another movie page I made 3 days ago is still functioning
Don't want to sound like I'm Tempting Fate, but it could've been worse, right?
Edited by RobertTYL on Jul 7th 2020 at 8:10:03 PM