-->''"Dude, we lost, like, everything. ;_;"''

January 12, 2008: Catastrophic hardware failure strikes the Wiki, resetting all articles back to mid-October. There was great lamenting and scrambling by editors to save articles from Google's and Yahoo's caches.

This is their story.

Not to be confused with the (obviously less important) Great Crashes of [[TheGreatDepression 1929]], [[TheGreatVideoGameCrashOf1983 1983]], [[UsefulNotes/TheGreatComicsCrashOf1996 1996]], 2007, [[TabletopGame/{{Shadowrun}} 2029 or 2064]].


The discussion below began in ContributorAnnouncements but was soon moved to and continued here.

The following pages were created and indexed here in the midst of the discussion and later deleted once the healing process had completed:
* [[http://tangent128.googlepages.com/fixer2.html HTML-to-Wiki format converter]]
* @@[=Images We Miss=]@@ - Images were not destroyed, but their [=URLs=] were forgotten. Here, tropers appended requests for anyone who might remember what/where particular images had been. (This is before the [[http://tvtropes.org/pmwiki/img_list.php Images List]] tool was created.)
* @@[=Pages Needing Formatting=]@@ - Index of pages whose content had been recovered, but badly--copy-pasting straight from caches and so forth. The HTML-to-Wiki converter was a huge help.
* @@[=Scavenger Hunt=]@@ - Index of pages containing the titles for all [=YKTTWs=] launched during the missing period, as well as the caches of various other index pages. Titles were blue-shifted as their caches were restored or crossed out if no cache could be found.
* @@[=Verify An Index=]@@ - Never really took off. %%
----
Tropers/FastEddie: The Great Crash of '08 update: We're back to life, of a sort, for tropes. No way to sugar-coat it: We lost just about everything from mid-December through 12JAN. Backups were corrupted. As of about 20 straight hours of being into trying to put things back together, I'm a getting a bit cross-eyed and am going to grab a few Z's. There may be hope for the lost material, after my brain comes back online.


Tropers/{{Tangent 128}}: Ouch. Yeah, I'd want a rest too.
What kind of corruption are we talking about? If it's some sort of "database is glitched but the raw data's there" thing, maybe some of us could volunteer to pick through the files and rebuild stuff by hand?

Or are we talking line-noise-bad?


Tropers/HeartBurnKid: Well, thanks for doing what you could... I think I'll spend some time tomorrow trawling through Google's cache and seeing if I can save some of my edits (I just did just that for the WCW page).


Tropers/{{Tangent 128}}: Oh, and insert RousingSpeech here. We will overcome! The DataVampires shall rue the day they made us enemies! And such.


Tropers/MorganWick: Shouldn't the backups be stored in a way that they wouldn't be affected by anything that would cause us to need them? Doesn't that defeat the point of backups? (Sorry, I love to nag)

You say mid-December, but everything in the Forums post-October is gone...


Tropers/HeartBurnKid: I'm wondering the same thing myself... please don't tell me the backup was stored on the server...

It looks like there was plenty of data loss for December and November on the wiki side, too; look at the Recent Changes page.


Tropers/{{Cassius335}}: Hell of a way to start a year. On reflection, only losing a month or so doesn't seem so bad. At least we didn't lose everything since the beginning.


Tropers/{{Lale}}: It sounds that way, but after browsing around, I found it amazing how much we write around here in a month...


Tropers/FastEddie: Certainly the best idea right now for restoring a given article is to grab it out of the Google cache. I'll keep jamming away at the backups to see what can be recovered, but it looks slow and if-fy at this point.

Tech-y stuff: The biggest operational error made here (by me) was to take the backups on a database level, rather than on a table-level. Some tables (the non-article tables, BBS, YKTTW, etc) live on one disk, the one that didn't fail, and the others (articles, history) live on the one that did. There was corruption in the history table. The whole-database backup would not-so-cleverly aggregate all the data into one corrupted store and move it off to a safe place. A frustrating thing is that the per-transaction storage (each edit/addition to an article) which could be used to roll the article as close to its latest version as possible is the thing that was corrupt. Really rotten luck.

Going forward, backups will be made on a table level, using two techniques (as SQL and as a binary copy -- we were just binary, before). Also, all transactions will be replicated to the new Search box so that there will be more backups per table to rely on.


Tropers/{{Kizor}}: Eddie! Good to have you here, those of us laboring with cache-pulling (see @@[=Scavenger Hunt Launches Part One=]@@) have been wondering about what you'd have to say. This is important: Is there any reason not to go all out and pull as many pages from the cache as possible? Can it interfere with retrieval or restoration on your end?


Tropers/SilentHunter: Nasty thing to happen. I'd help, but I don't know a thing about cache-pulling.

Where are our back-ups stored anyway?


Tropers/{{Kizor}}: Further, what word of the mid-October to mid-December changes? Does the devastation stretch across a month or a quarter of an year? Should we restore non-trope pages (such as MadeOfWin, avatar and the airbending etcetera) to where they were or under temporary names?


Tropers/{{Citizen}}: Yeah, I'm wondering when would be the time to start revising pages again. Should I be waiting for you to restore pages from backup first?


Tropers/FastEddie: Articles are pretty much as restored as they are going to get, so revisions can go forward. YKTTW and the Forum have a better chnace of being brought up to date by stuff I'm working on now. New entries and edits there are safe, but reconstruction-type stuff should hang on for a bit.

How to Cache-Grab: Search Google for the page name. If you search as a WikiWord, we're usually on the first page. Select the "cached" link on the item. That will take you to the text as it was a few days ago.


Tropers/{{Attilargh}}: Is there any way to get one's hands on the markup? Restoring pages like the series will otherwise be rather time-consuming.


Tropers/SilentHunter: Don't think so. Ignore formatting at the moment, IMO. Save our entries! Copy and paste the lot.


Tropers/{{Tangent 128}}: Restored as much of WorldCreationProject as I could. I'm pretty the missing pages were never filled in to begin with, but if you wrote something there, try to check.


Tropers/SilentHunter: Calling it a night now (it's 10.05pm in London). I'll be back tomorrow, but we might be done by then. Good work everyone!


Tropers/{{Adam850}}: Someone should edit the robots.txt file here to stop Google from cache-ing, so we can preserve the good cached pages.
Plus, Is there a way to look at a listing of the images that reside on the server, so we can find lost images?


Tropers/{{Tangent 128}}: OK! Progress on the formatting job! [[http://tangent128.googlepages.com/fixer.html I have a basic html-back-to-wiki thing going.]] Paste the html in, click the button, copy out. (Supports bold, italic, spoilers, and rules, but not colors, WMG, or categories. Also, links are translated as external links, not wiki words. Still, should be a big help.)


Tropers/{{Citizen}}: It would work much better if you ''didn't'' translate internal links as external. Leave off the http://www.tvtropes.org/pmwiki/pmwiki.php/Main/ part.


Tropers/MorganWick: That might be harder than you make it sound, just because of the extra coding needed to separate internal links from external, not to mention dealing with non-Main namespaces.

I just had a crazy idea: Would it be too insane to see if the [[http://www.archive.org Internet Archive]] could maybe lend us a hand? (Normally the Archive only allows access to archives more than six months old, but there's a part of me that thinks they just might be willing and able to lend us more recent files.)

How soon before YKTTW and the Forum are restored?


Tropers/{{Mith}}: I now have a local copy of the cached versions of everything that was in the category SpeculativeFictionTropes and its subcategories (so none of that is time-sensitive) and am working on more. Unfortunately Google doesn't like retrieving cached pages too fast, so it goes slowly. Move this to @@[=Scavenger Hunt=]@@ if you think it's more helpful there.


Tropers/FastEddie: No joy on restoring YKTTW or Forum data. Actually, we even lost the structure of the database tables for YKTTW. I've recreated the YKTTW structures (going backward from the SQL queries). The Forum data ... if it hadn't been running on a another box before this one, we wouldn't have what we've got. As far as restoration goes, then, this amazing pulling together of people to plunder the Google cache will have to suffice.

No kidding, folks, this effort to restore things that just ''sprang up'' among the Tropers ... just amazing. There's hope for the Internet, yet.

I'll be closing down the webserver at midnight to install the new backup goodies. Should take about 15 minutes or so.// later: That's installed.


Tropers/SilentHunter: HTML from cache doesn't translate for index mark-ups.


Tropers/{{Citizen}}: [[http://tvtropes.org/phpbb3/viewtopic.php?f=18&t=121&p=954#p954 Go to the forum and use my HTML pages]] to download all the cache for the Scavenger Hunt. I can't do it alone, Google keeps banning me for spam downloading...


Tropers/{{Medinoc}}: SoundEffectBleep! Google has "updated" HisDarkMaterials to the "lost" version, and I'm afraid many pages suffered that fate...


Tropers/{{Daviot}}: Good news, tropers. Most of my edits were before the archive date >_<;. Regardless, I've got caches of them, and will be fixing them shortly. {{Midonin}}, I only have the 7 Jan. Google cache for the R+V page. How do you want to go about tweaking it?


Tropers/{{billythehick}}: January 12th was my birthday. Not the best of omens... Still, my sister holds the record, her birthday heralded the start of the gulf war.
happy birthday to me...


Tropers/JohnQuixote: It's amazing to see such a small group of people working together to save a great site. This community is so much better than some others I visit. I just wish I had known about this earlier so I could have helped more; it looks like most everything has been saved, and everything that hasn't is lost forever (or for at least half a year). But seriously, just saying, thank you, everyone, for not only bringing this amazing site back, but for setting a good example for the rest of the 'net.


Tropers/{{Demetrios}}: I spent three hours last night reviving the Nightmare Fuel pages. But it was worth it.


Tropers/{{Donomni}}: Adding to the watchlist doesn't seem to be working right at all... took me a few dozen tries to re-add ''VideoGame/FinalFantasyXI'' to my list again.


Tropers/InsanityPrelude: Brought back a few pages, but most of the work had been done already when I found out about it. Incidentally, the index markup wasn't always coming out right, even when I used the handy little dropdown list here instead of trusting to the HTML-to-Wiki converter.
''Amazing'' job. You guys have the patience of gods~


Tropers/LooneyToons: Has anyone had any success recovering any lost ''images''?


Tropers/{{Adam850}}: Is it possible to get a listing of the images directory?


Tropers/{{fleb}}: If you successfully find a Google or Yahoo! cache, you can view-source and search for the '

Tropers/Da_Nuke: I must say, I'm touched. After doing my part to restore this site to its old pre-crash glory, I found out that pretty much every single page has been restored. This is by far the best wiki in the universe. T_T ^_^

On a side note, as of right now I'm downloading an entire archive of the site with HTTrack. Every single page will be saved to my hard drive. Building a full backup is kinda hard because that means leaving my computer on for like 3 full days, and my parents are really fussy with the power bill, but I'll try my best. It's the least thing I can do for this site.


Tropers/{{Kizor}}: It's now January 17th, 3:21 AM by your puny standards of Pacific Standard Time, and for the first time since they were started I see no page relating to the repairs in the 100 most recent changes.


Tropers/{{Pteryx}}: Whew, there -- I've saved one index and its members, however small (ThisTropeNameReferencesItself). Goes to show the job's not quite done yet, though. Who else wants to @@[=Verify An Index=]@@?


Tropers/{{Donomni}}: Checked a bit: It seems the only articles I have problems putting on the watchlist are ones I had before the Crash on after the cutoff date. Tried clicking remove as well... no go. Any ideas?


Tropers/{{Po8}}: Please see my offer of assistance with offsite backup in the WikiTechWishList. I'm pretty confident I can set up something that will be pretty regular and pretty complete, with help from those "on the inside".


Tropers/{{Heatherly}}: I'm going through to try and add any edits I'd contributed (I tend to have them betaread in [=IMs=] before pasting them here, and my [=IM=] program automatically logs any [=IMs=] I make). I wanted to help add entries, but looks like almost everyone had gotten that covered! I found that tropes I've written (which I had backed up on my own hard drive) were already back before I even knew they were missing. This was so incredibly awesome. I checked the page histories so I could credit the restorers on my contributor page.


Tropers/CodeMan38: Something needs to be done about MoodDissonance and SoundtrackDissonance, but alas, I'm too much of a GeniusDitz to take the task on myself without screwing it up somehow. The former, if I recall, was renamed to the latter so that the former could become more generalized, but we currently have two divergent versions of the same article...


Tropers/SilentHunter: When can we start getting stuff from archive.org?

Tropers/AnsemPaul: Oh man i feel so silly... I thought someone had removed my edits from some stupid reason agisant me, so I yelled at them. Opps.
I guess the universe doesn't revolve around me...

%%Meocross: wow did this just happen about a week ago!? just who kicked the database off the table?
%%
%%LooneyToons: Um, no, it happened about five months ago.
----