Follow TV Tropes

Following

Converting all data to UTF8 (now complete)

Go To

Hey tropers, we are about to undergo a large update to the tvtropes' database to convert the encoding. During this time the login system will be offline. You'll still be able to read TVTropes but you won't be able to post or edit anything during the conversion.

How long will the login system be down?

My most recent test run took 9 hours (my first test took 20+ hours so this is after a lot of optimizations to get down to 9 hours). ***it will now take 11 hours. see update below

Can you give me more details?

TVTropes was originally hosted on Windows' servers back in the day and all content on the site is encoded with Windows 1252 (a superset of ISO-8859-1, aka latin1).

According to the W3Tech, only 1.3% of internet traffic is ISO-8859-1. I'm guessing a large amount of that is TVTropes considering we have millions of pages defined with that charset.

The majority of the internet is encoded in UTF8. By doing this conversion it will bring us up to modern standards with the rest of the web and allow us to more easily support other languages and icons. It will also allow us to use more modern tools to help with editing such adding a WYSIWYG editor option. It will also help with code development as we often have to add special workarounds to continue to support this long deprecated charset.

Are we changing anything else?

While we have the login system offline, we are going to upgrade the edits history database table to include a sequence number. We've always wanted to do this but it could not be done while the site was online. That table has 40M+ rows of data. With this change we'll be able to make it so we can easily jump to any page when filtering to edits from a specific page or from a specific user. We have this for the forum and that's why you can jump to page 500 for example of a long running thread and not have a long delay.

When will this happen?

UPDATE: (take two) This is now scheduled for Mon Dec 5th at 8:30PM PST until Tue Dec 6th at 7:30AM

At 8:30PM PST you will be logged-off of TVTropes so make sure to save anything before that time. Once the conversion is done the next morning, I'll point the code to the new server and everyone should be auto logged in. If not, when you see the all clear announcement at the top of the page, go ahead and try logging in again.

The process will take approximately 11 hours. I had it down to 9 hours until I found examples of utf8 encoded values inside latin1 columns so I had to add some extra testing to ensure that data doesn't get double encoded in the process.

If you have any issues during the migration please send an email to thestaff@tvtropes.org

UPDATE: (complete) The migration is officially complete. It ended up taking nearly 12 hours but we got there. All data on TVTropes (hundreds of millions of rows of data) has now been converted from Latin1 to UTF8. We now have the same encoding as 99% of other websites and can support special characters and other languages. It will also allow us to build other tools such as a WYSIWYG editor.

Edited by itcdr on Dec 6th 2023 at 8:48:40 AM

TheTropper Good freakin' griefer from Allentown (Unitroper) Relationship Status: On the fine, fine line
Good freakin' griefer
#301: Dec 7th 2023 at 6:05:37 AM

About the monospace font, on the history page when it's on, I've noticed there's a difference in line spacing. Here's a comparison when it is off. And (if needed) here's how the spacing looked before the update (via WayBack). It is possible for the spacing to be fixed?

Your receipt. You can keep the freakin' change.
NeoChaos from San Jo you know Since: Jan, 2015 Relationship Status: Owner of a lonely heart
#302: Dec 7th 2023 at 9:46:02 AM

Minor nitpick about monospace font: can it use the browser-defined font for monospace instead of forcing Courier New? I prefer Consolas and I'm not fan of my option being overridden by the site. (Strangely, the post editor shows my preferred font just fine; this mostly affects the article editor and histories).

skewview Since: Jun, 2013
#303: Dec 8th 2023 at 12:05:51 AM

[up],[up][up] This discussion thread seems to have dwindled, best course of action now would be to make wishlist and bug reports.

AFK with issues, will return
JHD0919 One-Track Mind (he/him) from a 12-pack of Diet Coke (Troper in training) Relationship Status: Abstaining
One-Track Mind (he/him)
#304: Dec 8th 2023 at 10:01:46 AM

[up]Yeah, I concur. I therefore propose this thread is unstickied and locked due to having served its purpose.

This is Idol Tap. (My Troper Wall)
Lymantria Tyrannoraptoran Reptiliomorph from Toronto Since: Apr, 2015 Relationship Status: Historians will say we were good friends.
Tyrannoraptoran Reptiliomorph
#305: Dec 8th 2023 at 12:48:31 PM

Are the new history and edit fonts here to stay?

Join the Five-Man Band cleanup project!
kory Admin from a universe without doors (The New Guy) Relationship Status: watch?v=dQw4w9WgXcQ
Admin
#306: Dec 8th 2023 at 12:57:48 PM

[up] Yes, that’s why we added the monospace font toggle so anyone can change it back to the way it was.

Now monitoring Wishlist and Bugs
Lapsem Since: Aug, 2020
#307: Dec 8th 2023 at 8:46:38 PM

[up]I like that; that it gives people options. It also seems like it could be the first step towards that fancy visual editor you've been talking about.

Check out my Pantheon Search Database prototype!
kory Admin from a universe without doors (The New Guy) Relationship Status: watch?v=dQw4w9WgXcQ
Admin
#308: Dec 8th 2023 at 8:56:19 PM

[up]The visual editor is actually complete except for a few issues with the mark-up conversion we are trying to fix at the moment before we launch it for the engineers and moderators to test. After a short testing period then it’ll go live for everyone smile

Now monitoring Wishlist and Bugs
bwburke94 Friends forevermore from uǝʌɐǝɥ Since: May, 2014 Relationship Status: RelationshipOutOfBoundsException: 1
Friends forevermore
#309: Dec 9th 2023 at 3:57:55 AM

The "edit page" URL has included the term WYSIWYG since the beginning. Almost twenty years later, it's time to turn this into reality!

I had a dog-themed avatar before it was cool.
Lymantria Tyrannoraptoran Reptiliomorph from Toronto Since: Apr, 2015 Relationship Status: Historians will say we were good friends.
Tyrannoraptoran Reptiliomorph
#310: Dec 9th 2023 at 1:44:57 PM

How do I toggle the history/edit fonts again?

[down] Thanks, I just didn’t think to look there for some reason.

Edited by Lymantria on Dec 9th 2023 at 6:42:55 PM

Join the Five-Man Band cleanup project!
kory Admin from a universe without doors (The New Guy) Relationship Status: watch?v=dQw4w9WgXcQ
Admin
#311: Dec 9th 2023 at 1:46:43 PM

Toggle the Monospace Font option in your profile settings or on any editing page and that will change the edit history as well.

Edited by kory on Dec 9th 2023 at 1:48:08 AM

Now monitoring Wishlist and Bugs
Lymantria Tyrannoraptoran Reptiliomorph from Toronto Since: Apr, 2015 Relationship Status: Historians will say we were good friends.
Tyrannoraptoran Reptiliomorph
#312: Dec 10th 2023 at 7:33:15 PM

There’s no toggle on the history pages, though. Is this intentional?

Join the Five-Man Band cleanup project!
JHD0919 One-Track Mind (he/him) from a 12-pack of Diet Coke (Troper in training) Relationship Status: Abstaining
One-Track Mind (he/him)
#313: Dec 12th 2023 at 7:37:43 AM

[up]Personally, I don't mind. I think the edit history pages are easier to read now.

This is Idol Tap. (My Troper Wall)
AbsoluteRainbow Absolute Rainbow & the tales between worlds from Hanoi, Vietnam Since: Jul, 2023
Absolute Rainbow & the tales between worlds
#314: Dec 13th 2023 at 1:50:34 AM

What does WYSIWYG mean?

Absolute Rainbow
Amonimus the Retromancer from <<|Wiki Talk|>> (Sergeant) Relationship Status: In another castle
the Retromancer
#315: Dec 13th 2023 at 1:53:09 AM

[up] What You See Is What You Get, aka Visual Editor.

As asked before, is there anything left on the conversion topic?

TroperWall / WikiMagic Cleanup
AbsoluteRainbow Absolute Rainbow & the tales between worlds from Hanoi, Vietnam Since: Jul, 2023
Absolute Rainbow & the tales between worlds
#316: Dec 13th 2023 at 2:09:41 AM

What's the specific benefits of, well... WYSIWYG?

Absolute Rainbow
Amonimus the Retromancer from <<|Wiki Talk|>> (Sergeant) Relationship Status: In another castle
the Retromancer
#317: Dec 13th 2023 at 2:13:04 AM

[up] You see the page as it would appear on-page while typing, without pressing Preview every time. Microsoft Word works the same way as fonts are actually also invisible-to-users blocks.

TroperWall / WikiMagic Cleanup
alnair20aug93 🍊orange fursona🧡 from Furrypines (Long Runner) Relationship Status: Chocolate!
🍊orange fursona🧡
#318: Dec 20th 2023 at 11:58:51 PM

Wikipedia links with accented marks still has this jumbled mess

ᜇᜎᜈ᜔ᜇᜈ᜔|I DO COMMISSIONS|ᜇᜎᜈ᜔ᜇᜈ᜔
Amonimus the Retromancer from <<|Wiki Talk|>> (Sergeant) Relationship Status: In another castle
the Retromancer
#319: Dec 21st 2023 at 12:08:32 AM

Examples? Links usually have own URL Encode method because unicode in web address actually uses own format.

TroperWall / WikiMagic Cleanup
alnair20aug93 🍊orange fursona🧡 from Furrypines (Long Runner) Relationship Status: Chocolate!
🍊orange fursona🧡
#320: Dec 21st 2023 at 12:58:45 AM

I was editing the page for Charles Perrault, about his membership in the Academie Francaise, and the Wikipedia link almost ended up like this -> https://en.wikipedia.org/wiki/AcadA9mie_FranA7aise

I just ended up with the accented versions of the links.

ᜇᜎᜈ᜔ᜇᜈ᜔|I DO COMMISSIONS|ᜇᜎᜈ᜔ᜇᜈ᜔
AudioSpeaks2 He/Him (Greenhorn) Relationship Status: Oh my word! I'm gay!
He/Him
#321: Dec 21st 2023 at 1:00:41 AM

On Blackmail there's this entry for Hamilton linking to the Reynolds affair Wikipedia page

  • The Reynolds Affair drives much of the second act of Hamilton. Maria Reynolds seduces him, then she and her husband blackmail him by threatening to publish the proof. Which is a problem, given that Hamilton is 1) active in politics and 2) mostly-happily married. Unlike most examples of this trope, Hamilton does go public, though only because the proof of his affair exonerates him from an accusation of embezzlement.note  It works, but he almost loses his wife and his career as a result.

Yet it ends up saying "Bad Title"

Art Museum Curator and frequent helper of the Web Original deprecation project
Twiddler (On A Trope Odyssey)
#322: Dec 21st 2023 at 6:32:44 PM

[up][up], [up] I replaced the links. They work now.

Edited by Twiddler on Dec 21st 2023 at 6:34:07 AM

AudioSpeaks2 He/Him (Greenhorn) Relationship Status: Oh my word! I'm gay!
He/Him
#323: Dec 21st 2023 at 6:35:14 PM

We should probably go through We Are Not Alone Index and fix as many links as possible

Art Museum Curator and frequent helper of the Web Original deprecation project
AbsoluteRainbow Absolute Rainbow & the tales between worlds from Hanoi, Vietnam Since: Jul, 2023
Absolute Rainbow & the tales between worlds
#324: Dec 21st 2023 at 6:35:52 PM

When will WYSIWYG be added?

Absolute Rainbow
kory Admin from a universe without doors (The New Guy) Relationship Status: watch?v=dQw4w9WgXcQ
Admin
#325: Dec 21st 2023 at 6:43:06 PM

[up]The goal is to hopefully have it live before the end of the year. 🤞

Now monitoring Wishlist and Bugs

Total posts: 331
Top