Follow TV Tropes

Following

TV Tropes was offline due to server issues! Editing has been restored.

Go To

As many of you reported TV Tropes went offline at 2:40am EST on July 7th. It was offline for over 14 hours. The worst outage in many years. We did receive email and text alerts when it happened but unfortunately it was a major hardware failure which took quite a while to get under control.

The cause was a total failure of our database cluster. 6 of our 8 hard drives failed simultaneously resulting in a complete loss of data. We had to have our server company replace the cluster and then we had to rebuild the site from database backups. We do automatic backups every morning. Unfortunately the failure happened hours before the next backup so 24 hours of changes were lost.

To make it worse the history of wiki changes is only updated twice a week because it is over a 1TB in size. We are working on restoring that now so the history tab is blank on all pages until it's done. Editing will be offline for another 24 hours until we get that fixed. And it means we'll lose 72 hours of wiki history due to the timing of the last backup.

We will be working on optimizing our database structure so we can increase the frequency of our database backups to protect the data in the future.

We have redundant web servers on a load balancer, redundant database servers in a cluster and redundant hard drives in every server. So how did this happen? According to our server company there was a manufacturers bug in the firmware of the specific model that 6 of our 8 hard drives were on. That bug caused the disks to die after a certain number of hours running. We don't yet have all the details. They are reaching out to the manufacturer to get more information. I'll update here as I learn more.


UPDATE: (July 8th)

Editing is now enabled! History should be restored as of July 4th 10am EST. The history database imported faster than I expected (4 hours to decompress 1.1TB sql file, 12.5 hours to import)

The only thing I haven't done yet is purge the CDN cache. You must logout to view a page cache. Logged-in users get the live site. Not all pages are still cached and they do expire.

I'll hold off on purging the cache for a few more hours. If there is some specific edit you remember doing during those 24 hours that were lost you may be able to find it by logging out and viewing the cached page. Then login and make that edit again.

Edited by itcdr on Jul 8th 2020 at 7:12:45 AM

Ai-Fan from Germany Relationship Status: I'm just a poor boy, nobody loves me
Jul 8th 2020 at 1:24:41 PM

Thank God! I've missed you, guys![tup]for the hardworking people who fixed it.

writer/producer/director/visionary/troper Ai-Fan The Master Of The Romantic (But Deadly) Drinking Game And Haiku™
Jul 8th 2020 at 1:46:08 PM

The sextuple cock-up... Now that's a name.... tongue

Glad the site is back! It makes our daily routine better :)))

- Fly, robin, fly! - ...I'm trying!
jaketroper Using the good old jellyfish from New York, New York Relationship Status: [TOP SECRET]
Using the good old jellyfish
Jul 8th 2020 at 3:37:56 PM

Phew! It could have been way worse! Thanks to the guys at TVT for fixing this!

TV Tropes is enhancing my life. It will enhance yours, too!
WarJay77 Warren Kidd from Upstate New York Relationship Status: Armed with the Power of Love
Warren Kidd
Jul 8th 2020 at 3:58:07 PM

I hope we can figure out where the videos went. I uploaded two of them Monday night before the server crash; they both went through. Now I'm stuck with the 10% glitch again and it sucks because the videos were properly uploaded and now they're refusing to upload again. If there's any chance of getting the information from Vimeo and restoring them, that'd be awesome.

3 stories in the works, more on my profile if you're curious - Tropes Needing TRS - Trope Idea Salvage Yard
Jul 8th 2020 at 4:48:01 PM

Also, what was with the Wiki briefly coming back blank?

Complete Monster quotes cleanup
Crossover-Enthusiast Happy Bi Day! from quarantine (send Red Lobster biscuits) Relationship Status: Chocolate!
Happy Bi Day!
Jul 8th 2020 at 5:27:09 PM

Is the save profile save button not supposed to be there?

Shit, let's be Disobedient
jaketroper Using the good old jellyfish from New York, New York Relationship Status: [TOP SECRET]
Using the good old jellyfish
Jul 8th 2020 at 7:02:53 PM

Oh yeah, I saw the temporary restoration. Nothing existed. No articles, or forum edits, or even accounts. It was so weird...

TV Tropes is enhancing my life. It will enhance yours, too!
Noah1 Noah 1 from Somewhere Relationship Status: THIS CONCEPT OF 'WUV' CONFUSES AND INFURIATES US!
Jul 8th 2020 at 9:13:05 PM

Well everyone, LEGO Rewind is finally back on TV Tropes! Again, this couldn't have been accomplished without Albert 3105's discovery of the cache exploit and screen-capping, so special thanks. Any and all Wiki Magic and crosswicking is more than welcome.

Any decent person should have the right to live and do as they please as long as it's functional and harmless. Simple as that.
Jul 8th 2020 at 9:36:31 PM

I had freaked out a bit. Thought I was hacked. Then I tried to go to this site on my phone which I had never logged on with. Said it couldn't reach the server. I instantly felt relief but I wondered what happened. Glad to see it's over now.

KoffingMugg Creator of Universes from Edge of Time
Creator of Universes
Jul 8th 2020 at 10:14:22 PM

I honestly freaked a bit when this happened. I thought there was something wrong with my connection or my computer/phone. I was lucky to only loose a minute's worth of editing, but my heart goes out to everyone who lost hours' worth of research and editing to what I'm going to call The Day the Tropes Died.

I am the captain of my soul. - Invictus
Unicorndance Logic Girl from Thames, N.Z. Relationship Status: Above such petty unnecessities
Logic Girl
Jul 8th 2020 at 10:42:00 PM

That's a bit of a morbid name. I was thinking some names for the event could be:

  • The 2020 Glitch
  • The Not-So-Great Crash
  • The Hard Drive Incident

For every low there is a high.
JXZ entry pimp walk
entry pimp walk
Jul 8th 2020 at 10:50:57 PM

[up]

  • The Sextuple Cock-Up
  • Tropes Machine Broke
  • The Fairly Bad Crash Of 2020
  • Withdrawal Day
  • C-C-C-C-C-Combo Crash

Edited by JXZ on Jul 8th 2020 at 1:54:16 PM

my brain is a computer with 4k of ram
WarJay77 Warren Kidd from Upstate New York Relationship Status: Armed with the Power of Love
Warren Kidd
Jul 8th 2020 at 10:50:59 PM

"Those Two Heroic Hard Drives" Day

3 stories in the works, more on my profile if you're curious - Tropes Needing TRS - Trope Idea Salvage Yard
Piterpicher A child of the 2000s, for better or worse from Poland, for real Relationship Status: All is for my lord
A child of the 2000s, for better or worse
Jul 8th 2020 at 10:53:36 PM

I prefer "The Unlucky 7/7 Day", to be honest, or something similar (some may suggest something based on Persona 5 as it's the in-game day Atlus initially blocked streaming from due to spoiler stuff, but IDK).

Edited by Piterpicher on Jul 8th 2020 at 7:57:56 PM

It's the final year of school (assuming I don't go to another one) and exams are looking on the horizon, so I may not be as active.
mightymewtron Troper Fangirl Relationship Status: Hugging my pillow
Animefreak1290 One of the strangest human beings you will ever from Court of Azathoth Relationship Status: Above such petty unnecessities
One of the strangest human beings you will ever
Jul 8th 2020 at 11:25:36 PM

Saving the articles will restore them. I tried with Magia Record’s Nightmare Fuel and Tear Jerker pages (which got wiped completely) and saving them restored the pages. I’d recommend hunting down the pages that were wiped and saving them to restore them.

Just a guy on TV Tropes who loves Anime and Video Games. Savy?
Jul 9th 2020 at 1:28:44 AM

We should really look into archiving this site somewhere, to ensure there's no chance of the website vanishing without a backup/

bwburke94 My Ice Cream Waifu from Remnant Relationship Status: Longing for my OTP
My Ice Cream Waifu
Jul 9th 2020 at 5:36:38 AM

If we absolutely have to give this a name, "The Crash of 2020" probably works best. It's not a Great Crash.

Misspelled trope names dealt with... for now. Moving back to the Understatement project soon?
Crossover-Enthusiast Happy Bi Day! from quarantine (send Red Lobster biscuits) Relationship Status: Chocolate!
Happy Bi Day!
Jul 9th 2020 at 6:42:44 AM

Nope, six of TVTropes' eight hard drives all just fucked off into the sunset simultaneously. pardon the language

Anyway, as a name for this endeavor, I like "The Unlucky 7/7 Day" and "C-C-C-C-C-Combo Crash".

Shit, let's be Disobedient
Jul 9th 2020 at 6:48:55 AM

Oh, I understand. I just got confused because the same day, a BTS album release crashed Twitter (no wonder why they have their own entry on Demand Overload), but it turns out it was hardware, like what commonly happens to the Internet Archive. Luckily, the only edit I made was to Screwed by the Network concerning how the COVID-19 pandemic affected Broadway. Also, when I re-added that, I added an example about a similar incident (the revival of The Rocky Horror Show ending due to declining tourism after September 11th.)

wingedcatgirl 🐱🎶 from a cat cafe Relationship Status: Crazy Cat Lady
🐱🎶
Jul 9th 2020 at 8:57:57 AM

That's because the page title is 77 characters long. I forget the limit, but it's definitely less than that.

Sylvi& | it/they/she | autistic plural trans catgirl | Hang out with us on Discord if you wanna
Piterpicher A child of the 2000s, for better or worse from Poland, for real Relationship Status: All is for my lord
A child of the 2000s, for better or worse
Jul 9th 2020 at 8:58:05 AM

Again, too long title, not the crash. It's 77 chars, the limit's 64. I recommend removing the episode title.

It's the final year of school (assuming I don't go to another one) and exams are looking on the horizon, so I may not be as active.

Total posts: 295
Top

How well does it match the trope?

Example of:

/

Media sources:

/

Report