Hey tropers, we are about to undergo a large update to the tvtropes' database to convert the encoding. During this time the login system will be offline. You'll still be able to read TVTropes but you won't be able to post or edit anything during the conversion.
How long will the login system be down?
My most recent test run took 9 hours (my first test took 20+ hours so this is after a lot of optimizations to get down to 9 hours). ***it will now take 11 hours. see update below
Can you give me more details?
TVTropes was originally hosted on Windows' servers back in the day and all content on the site is encoded with Windows 1252 (a superset of ISO-8859-1, aka latin1).
According to the W3Tech, only 1.3% of internet traffic is ISO-8859-1. I'm guessing a large amount of that is TVTropes considering we have millions of pages defined with that charset.
The majority of the internet is encoded in UTF8. By doing this conversion it will bring us up to modern standards with the rest of the web and allow us to more easily support other languages and icons. It will also allow us to use more modern tools to help with editing such adding a WYSIWYG editor option. It will also help with code development as we often have to add special workarounds to continue to support this long deprecated charset.
Are we changing anything else?
While we have the login system offline, we are going to upgrade the edits history database table to include a sequence number. We've always wanted to do this but it could not be done while the site was online. That table has 40M+ rows of data. With this change we'll be able to make it so we can easily jump to any page when filtering to edits from a specific page or from a specific user. We have this for the forum and that's why you can jump to page 500 for example of a long running thread and not have a long delay.
When will this happen?
UPDATE: (take two) This is now scheduled for Mon Dec 5th at 8:30PM PST until Tue Dec 6th at 7:30AM
At 8:30PM PST you will be logged-off of TVTropes so make sure to save anything before that time. Once the conversion is done the next morning, I'll point the code to the new server and everyone should be auto logged in. If not, when you see the all clear announcement at the top of the page, go ahead and try logging in again.
The process will take approximately 11 hours. I had it down to 9 hours until I found examples of utf8 encoded values inside latin1 columns so I had to add some extra testing to ensure that data doesn't get double encoded in the process.
If you have any issues during the migration please send an email to thestaff@tvtropes.org
UPDATE: (complete) The migration is officially complete. It ended up taking nearly 12 hours but we got there. All data on TVTropes (hundreds of millions of rows of data) has now been converted from Latin1 to UTF8. We now have the same encoding as 99% of other websites and can support special characters and other languages. It will also allow us to build other tools such as a WYSIWYG editor.
Edited by itcdr on Dec 6th 2023 at 8:48:40 AM
This is going to be so much fun.
pokes stick
yay we're alive!
Edited by Ookamikun on Dec 6th 2023 at 7:28:54 PM
totally non-filler title
Officially intimidated
A bit late because I have just returned, but congrats for the conversion! ![[tup]](https://static.tvtropes.org/pmwiki/pub/images/thumbs_up_emote.png "[thumb up]"){kind=small}
Most likely 'to be'
YES! Emojis work normally now so I can complete my stinger!
[Insert clever saying here]
Hello
Not to surprise that converting a massive amount of code to UTF 8, when this website has like, errr, over 600k pages (I hope I got it right).
Anyways, the additional features should serve this website for convenience of its users 
Oh ok. I'm really bad at anything related to math so.
Logic Girl
Sorry for any inconvenience, but... what's exactly changed? The only thing I notice that's different is an ability to change the font on the edit screen, and that'd be weird if it was the only change.
For every low there is a high.
Big Catch, Sparkle Edition
We can use non-Roman and accented characters without fancy markup.
Currently Working On: Incorruptible Pure Pureness
the Retromancer
Nobody addressed it, but what about the secondary announced thing about giving all edits their numbers? I've assumed it means fixing troper's edit histories to include edit id, but this doesn't seem to be the case.
Edited by Amonimus on Dec 6th 2023 at 7:38:57 PM
TroperWall / WikiMagic Cleanup
the Retromancer
![[up]](https://static.tvtropes.org/pmwiki/pub/smiles/arrow_up.png "[up]"){kind=icon}
Meaning it's an internal setup and the related fixes will come later?
Btw I think the pinned post could use a tl;tr versiom summary since people keep asking what's different, and dunno if everything is working it could get locked in a day or two.
TroperWall / WikiMagic Cleanup
Admin
Yessir the changes will come later. Probably not too long after we get the visual editor
working, that’s right after most of the editing improvements are pushed up.
Edited by kory on Dec 6th 2023 at 9:03:31 AM
Now monitoring Wishlist and Bugs
Aztec God of Corn
So what exactly caused this exactly? I'm not active in the forums so I'm out of the loop, did someone try to attack the site or something? I recall the whole thing with 4chan and snoot game a short while back, related or am I just paranoid?
Life is but a mask worn on the face of death
the Retromancer
Are you at right thread? We did cut Snoot Game for harassment issues, but there is no relation to... allowing pasting non-English letters to articles, which is in turn necessary for the Visual Editor.
Edited by Amonimus on Dec 6th 2023 at 10:14:45 PM
TroperWall / WikiMagic Cleanup
Flashgitz's Worst Enemy... (He/him)
The header probably has a better explanation than what I can tell you, but I swear we weren't attacked; there was a need to convert our website to UTF 8, and the ability for tropers to be logged in had to be briefly disabled during the conversion.
Again, you have nothing to be worried about right now.
Edited by OmegaPC777 on Dec 7th 2023 at 2:05:02 PM
"How could you stop an idea?" (Check out my troper wall if you can!)
The layman's explanation is that the foundation of the site was updated to comply with modern web standards and make it easier to do cool stuff in the future. We won't see any changes right away, this is setting things up for later.
(I used to do this stuff for a living, the technobabble was easy to understand)
Edited by rjung on Dec 6th 2023 at 11:43:50 AM
Out the gate, the biggest advantage right now is for doing non-English language editing. Though, I've not done extensive tests, it looks like automatic translations work better, as do some accessibility options (screen readers, text to speech, etc.).
AFK with issues, will return
Rookie Trope Repairer
Unpopular opinion, but I actually prefer editing in Arial font, especially in Dark Mode. The font is more legible to me than monospace, and it's given me a few less headaches. It also reflects how the text will look on the article without having to preview it, so I can save that feature for double-checking markup.
"As long as I have my comrades with me, I can do anything!" (She/Her) (Current Focus: Cleaning Hell Is That Noise misuse)
I believe that is why they made a toggle, though the markup in proportional text has problems with undistinguishable quotemarks.
AFK with issues, will return
Rookie Trope Repairer
Good point, although that can easily be taken care of thanks to the toggle.
Redmess
And to be clear, the two outages of half a day were planned and part of this update (the first one was a false start).
Optimism is a duty.
the Retromancer
I wouldn't called the second one planned, more like the first one didn't work so it was attempted again later that day.
TroperWall / WikiMagic Cleanup
Redmess
It was still planned in the sense that a new update was planned as soon as the first one failed. It was still intended to happen.
Optimism is a duty.
Oh ok I’ll remove that one.
Now monitoring Wishlist and Bugs