Encoding errors (fix in progress)

[mark]

Hi all,

First of all, no data has been lost. We can fix the encoding issues you're seeing, but it will take me some time to write and validate a script to do so. I'm working on this now.

Updates

• I have completed writing the script that will fix things. I've tested it on a few specific accounts and validated that it works.
• I'm now running it on fail_fandomanon which... sorry y'all, your community is chonky and this will take some time. The script has to go clear out bad cache entries in MemCache on top of re-migrating the 13,548,827 comments, so it will be a few hot minutes.
• I'm going to start running the script on the other ~6,000 accounts that were impacted. This will take a few hours end to end, and I can't predict when any specific account will get fixed. I'll update when they're all done though.
• All accounts should be fixed. Except FFA, sorry! Y'all are still chugging. Please let me know if you see anything still broken.

Again, sorry for this. Thanks for your patience while we sorted it out.

What Happened

Today we've been doing some database migrations in preparation for an upcoming upgrade, and unfortunately we discovered an incompatibility in one of our systems that caused some errors in the migration. This resulted in some of the migrated entries and comments being copied over incorrectly which is resulting in the weird text you're seeing.

The migration process maintains a copy of all of the existing data that was migrated, in the original form, so we have the original content and can go fix it. This is going to be a somewhat manual process though, as I will have to do some copying of content from one place to another. (Basically, a re-migration, but without the thing that caused the original issue.)

I'll update this post when the problem is fixed. And, as usual, apologies for the issue. When I was testing this last night on my own journal and Dreamwidth's journals, it worked flawlessly -- because I was only running it on one of our migration hosts -- and the problem happened on host #3 (which is a newer host that is running a newer version of Perl.)

Comments

[katherine]

Thank you for the information and plan! The error's made for a surreal reading (and not reading) experience.

[sky]

Thank you for the update!

[dewline]

Good to know the work's in progress!

[ride_4ever]

My TYK icon smiles in recognition of your TYK icon.

[morrow]

Thank you for the speedy update!

[sixbeforelunch]

Thank you for the update! :)

[ride_4ever]

Thank you kindly for the update and for EVERYTHING that Dreamwidth is!

[veritas_poet]

As always, thanks to you & the team for all your work!

[ex_flameandsong751]

Thank you kindly for the hard work you do to keep this site running, especially on the weekend.

[healyg]

Good to know it's being worked on!

[bluedreaming]

Thank you!

[etfb]

Ƥēŕḟḕƈțƚƴ ṷṅđĕȓșʈɑɲɖȧɓļę. Ħăƥᵱḙṅṧ ấ ĺōţ.

[mark]

:dead:

[paserbyp]

Maintaining legacy Perl code became harder due to fact that language is dead for many years. How will be difficult to migrate to a more modern language?

[o_klyueva]

perl is not dead. two new versions were released this year :
5.37 4 months and 3 weeks ago (27 May 2022)
5.36 4 months and 3 weeks ago (27 May 2022)

[alexanderr]

> ...a newer version of Perl...

I see!

[toaster_hacker]

It's downright awe-inspiring how quickly y'all identified the issue, wrote a script to help fix it, and already began implementing it. You're all incredible, thank you for the amazing work!

[yuuago]

Thanks for the info! It was pretty weird to see stuff all scrambled like that. Glad it was a relatively easy fix. <3

[hokuton_punch]

Thank you so much for figuring it out! ♥

(13.5 million comments - d a m n...)

[andraste]

I'm now running it on [community profile] fail_fandomanon which... sorry y'all, your community is chonky and this will take some time. The script has to go clear out bad cache entries in MemCache on top of re-migrating the 13,548,827 comments, so it will be a few hot minutes.

... wow, that is a *lot* of arguments about slingbabies and onions. Well done, meme!

[sunnymodffa]

We appreciate your hard work!

[denise]

Sorry again about using your community as an inadvertent porthole to an eldritch dimension! We swear we'll clean up the sulfur and the ectoplasm by the time we're done...

[silverflight8]

Thanks for fixing! And I'm lol-ing at the comment that meme is chonky. Holy crap I hadn't realized how many comments there were, accurate.

Also I must say it is kind of hilarious to browse meme at the moment. It looks like cthulhu posted comments but other people are responding normally and being like "oh yes, I really liked xyz" lmao to complete eldritch horrors of comments.

[mark]

Haha yeah. The specifics of the problem mean that short comments wouldn't be impacted -- since they're too small to compress. It's only when they get long enough to be worth compressing that the elder gods want to have a word!

[lb_lee]

*opens door* hey guys, what'd I miss--oh. Pardon me. *Retreats, gently shuts door behind me*

(That sounds like a nightmare, had no idea it was even happening.)

[stealthsystem]

LOL same here. I... kind of want to see some of those Eldritch comments to see how bad the screen reader would butcher them.

[nonnie009]

Thanks for all your hard work to keep dreamwidth running!

re: The script has to go clear out bad cache entries in MemCache on top of re-migrating the 13,548,827 comments, so it will be a few hot minutes.

13,548,827, wow! Ffa is still in Read-Only User state. I wonder how long it will take to restore everything like it was before.

[mark]

My estimate was 8 hours from the start, give or take a bit. I think it'll be done in the next hour or two. Sadly, the script doesn't print out a status percentage, and I don't want to stop it now to add one (or it'll reset.)

[falena]

Thank you for the update! Sorry, Mark, I'm sure this is not how you wanted to be spending your Sunday. Sending positive vibes your way.

[ordinary_joe_backup]

What a story.

Being involved in software development myself I understand how horrible it is to support a legacy system which is nearly quarter of century old.

Keep up the good work, DW is last LJ implementation worth mentioning.

It is like a piece of art and engineering which worth to be kept alive.

Original LJ is finally dead, servers in Moscow, censorship and russian war propaganda - that's all what left over there.

Clearly muskovites wont have data centers to run it in few years.

Thank you guys for a good work!

[paserbyp]

Perl not dead yet and LJ not dead also, because Muscovites will re-write LJ from Perl to new language Putin…

[evalerie]

I've seen too many migrations where nobody kept the original version of what was copied, and then something went wrong, and it was needed. Thank goodness you did!

Thanks for being there keeping everything going!

[kikibug13]

I don't know if the comm issue is related to this problem, but I thought it might be worth dropping a line - currently bakerstreet seems to be read-only, no new entries or comments can be posted to it. As it is another rather large comm - 13,896,348 comments received - there was a thought that it might be connected.

Thank you so very much for your work!

[sockiness]

This started around 9pm PST and the comm itself still reads as open on the profile, so I'm thinking it might be a glitch? Not sure if it'll fix itself or if it needs help.