Encoding errors (fix in progress)

[mark]

Hi all,

First of all, no data has been lost. We can fix the encoding issues you're seeing, but it will take me some time to write and validate a script to do so. I'm working on this now.

Updates

• I have completed writing the script that will fix things. I've tested it on a few specific accounts and validated that it works.
• I'm now running it on fail_fandomanon which... sorry y'all, your community is chonky and this will take some time. The script has to go clear out bad cache entries in MemCache on top of re-migrating the 13,548,827 comments, so it will be a few hot minutes.
• I'm going to start running the script on the other ~6,000 accounts that were impacted. This will take a few hours end to end, and I can't predict when any specific account will get fixed. I'll update when they're all done though.
• All accounts should be fixed. Except FFA, sorry! Y'all are still chugging. Please let me know if you see anything still broken.

Again, sorry for this. Thanks for your patience while we sorted it out.

What Happened

Today we've been doing some database migrations in preparation for an upcoming upgrade, and unfortunately we discovered an incompatibility in one of our systems that caused some errors in the migration. This resulted in some of the migrated entries and comments being copied over incorrectly which is resulting in the weird text you're seeing.

The migration process maintains a copy of all of the existing data that was migrated, in the original form, so we have the original content and can go fix it. This is going to be a somewhat manual process though, as I will have to do some copying of content from one place to another. (Basically, a re-migration, but without the thing that caused the original issue.)

I'll update this post when the problem is fixed. And, as usual, apologies for the issue. When I was testing this last night on my own journal and Dreamwidth's journals, it worked flawlessly -- because I was only running it on one of our migration hosts -- and the problem happened on host #3 (which is a newer host that is running a newer version of Perl.)

Comments

[paserbyp]

I feel your pain and maybe re-writing legacy code not only one solution?

[denise]

Thankfully it only becomes a problem once every few years! Usually we catch these things ahead of time, but this time we just missed that the new database machine was running a different version of Perl and MySQL than we tested with.

[paserbyp]

Believe me it will be repeated over and over again with different scenario of upgrade and QA test failures.