Encoding errors (fix in progress)

[mark]

Hi all,

First of all, no data has been lost. We can fix the encoding issues you're seeing, but it will take me some time to write and validate a script to do so. I'm working on this now.

Updates

• I have completed writing the script that will fix things. I've tested it on a few specific accounts and validated that it works.
• I'm now running it on fail_fandomanon which... sorry y'all, your community is chonky and this will take some time. The script has to go clear out bad cache entries in MemCache on top of re-migrating the 13,548,827 comments, so it will be a few hot minutes.
• I'm going to start running the script on the other ~6,000 accounts that were impacted. This will take a few hours end to end, and I can't predict when any specific account will get fixed. I'll update when they're all done though.
• All accounts should be fixed. Except FFA, sorry! Y'all are still chugging. Please let me know if you see anything still broken.

Again, sorry for this. Thanks for your patience while we sorted it out.

What Happened

Today we've been doing some database migrations in preparation for an upcoming upgrade, and unfortunately we discovered an incompatibility in one of our systems that caused some errors in the migration. This resulted in some of the migrated entries and comments being copied over incorrectly which is resulting in the weird text you're seeing.

The migration process maintains a copy of all of the existing data that was migrated, in the original form, so we have the original content and can go fix it. This is going to be a somewhat manual process though, as I will have to do some copying of content from one place to another. (Basically, a re-migration, but without the thing that caused the original issue.)

I'll update this post when the problem is fixed. And, as usual, apologies for the issue. When I was testing this last night on my own journal and Dreamwidth's journals, it worked flawlessly -- because I was only running it on one of our migration hosts -- and the problem happened on host #3 (which is a newer host that is running a newer version of Perl.)

Comments

[katherine]

Thank you for the information and plan! The error's made for a surreal reading (and not reading) experience.

[sky]

Thank you for the update!

[dewline]

Good to know the work's in progress!

[morrow]

Thank you for the speedy update!

[sixbeforelunch]

Thank you for the update! :)

[ride_4ever]

Thank you kindly for the update and for EVERYTHING that Dreamwidth is!

[ride_4ever]

My TYK icon smiles in recognition of your TYK icon.

[veritas_poet]

As always, thanks to you & the team for all your work!

[ex_flameandsong751]

Thank you kindly for the hard work you do to keep this site running, especially on the weekend.

[healyg]

Good to know it's being worked on!

[bluedreaming]

Thank you!

[etfb]

Ƥēŕḟḕƈțƚƴ ṷṅđĕȓșʈɑɲɖȧɓļę. Ħăƥᵱḙṅṧ ấ ĺōţ.

[paserbyp]

Maintaining legacy Perl code became harder due to fact that language is dead for many years. How will be difficult to migrate to a more modern language?

[mark]

:dead:

[o_klyueva]

perl is not dead. two new versions were released this year :
5.37 4 months and 3 weeks ago (27 May 2022)
5.36 4 months and 3 weeks ago (27 May 2022)

[mark]

Ironically, the problem today is actually that Perl 5 is under active, fairly rapid development, and I unintentionally ran the migration script on a modern Perl version that expected Unicode strings.

But, the Dreamwidth code (based on the LiveJournal code) is so old that we use binary strings... because we gzip the entry text before inserting it into the database.

So, the modern Perl string handling was doing some kind of WTF-encoding because it didn't know what to do with the binary crap we were shoving into a thing that was expecting UTF8 (I'm assuming, I'm not entirely sure what it wanted.)

[azurelunatic]

It would probably be a many year project. And from some of the troubleshooting discussion, it looks to me that this was due to not a Perl problem, but a problem with different versions of MySQL.

Edit: Mark knows more than I do, refer to his comment above

[alexanderr]

> ...a newer version of Perl...

I see!

[denise]

The most recent .0 Perl version was released in May of this year; it's far from a dead language! There's no reason to rewrite the site in another language (a project that will, from decades of experience, never be completed and introduce eleventy billion new horrible bugs and security vulnerabilities) when there's no driving technical reason for it.

The extremely short explanation of this issue: LJ started accumulating user data in 1999, before Perl had any real Unicode support, and even after Unicode support was added in experimental fashion in 5.6 in 2000 it was half-baked until at least the release of 5.8 in 2002 and the horrible workarounds we-at-LJ had developed to handle text in encodings other than ISO-8859-1 functioned better than Perl's native Unicode support for a few years. That's long since changed, of course, but even after LJ upgraded the version of Perl it was using, there were another few years where it made more sense to keep supporting the horrible workarounds instead of deferring to Perl's native Unicode support because the conversion process would have been a nightmare. I don't know if you were on LJ circa 2004-2005, but there was a whole thing where people had to go run the text of their journal through the encoding converter or else everything displayed as ????? -- that was when we-at-LJ made the switch over, but we still had to keep some support for the old horrible workaround because there was a ton of unconverted text, some people's browsers didn't handle things correctly, and there were still some ways people could get non-Unicode encoded text to the site; it was a mess for a few years.

Fast forward to us starting DW in 2008, once Perl Unicode support was perfectly robust. We did a bunch of work to make sure there was no way anyone could get non-Unicode encoded text to us, but because we allow people to import their journals from LJ, some of that old data was submitted during the Years of Terrible Encoding Mess, and LJ's Unicode conversion tool didn't always catch everything, we couldn't 100% eliminate those old terrible workarounds and have to do some processing to catch them. That's the root cause of what happened today: the new database cluster was running a newer version of Perl and MySQL than the test database that Mark had been testing on. If we didn't have to keep around some of that support for terrible old encoding issues, it likely wouldn't have been an issue, but because we do have that legacy data that's more likely to have bad encoding, there's a certain amount of text encoding we have to handle manually in our code instead of passing it off to Perl and MySQL and just let them handle it. We accidentally didn't test against the current versions in our test environment, so stuff got mangled.

It's not the language's fault, in other words: it's a bunch of legacy decisions that were correct in 2000-2004 or so when we-at-LJ made them, have since been updated/modernized, but occasionally prove to be a massive pain now because we have to support 23 years of legacy data that is often a giant mess. The steps we have to take to keep that giant mess from being an even worse mess occasionally become a giant mess themselves, which is what happened this time.

(EDIT: Ha, like three people beat me! Sorry for the inadvertent spam.)

[mark]

Yeah I mean, I'm not entirely certain where in the chain of dependencies it broke. It could be Perl or MySQL at this point, as both got upgraded on the machine in question.

[denise]

oh god I'd forgotten about the gzip thing even over and above the support for old data thing

[denise]

Really, you can sort of just throw it all under the category of "we are hauling around 23 years' worth of data that is a lot messier than we'd like and we occasionally have to torture the technology in ways the creators never expected in order to make it work"

[alchemise]

Thank you for this explanation!

[dewline]

I know just enough about text formatting to understand how that could turn into a mess.

[toaster_hacker]

It's downright awe-inspiring how quickly y'all identified the issue, wrote a script to help fix it, and already began implementing it. You're all incredible, thank you for the amazing work!