dw_maintenance | Tuesday night importer update

Because computers are terrible, the thing we thought was going to be a relatively easy fix for large comment import jobs is (inevitably) turning out to be a lot more complicated, so we don't have any news for you there except "we're working on it, we think we're making progress, but we've thought that before and been wrong", sigh.

Anyone who I've told that they're on The List for us to run their import job manually: we're crunching through those; manual mode is just a lot slower than the usual automated system and the nature of the problems mean we have to stop and restart the jobs pretty frequently. I will catch up with everyone whose jobs we've managed to crunch through tomorrow or whenever we make it through the full list, whichever works out best for time management purposes! If you have a job that's failed comment imports more than 10 times, let us know the name of the account at support@dreamwidth.org and I'll put it on The List.

Thank you all very much for your patience while we work to wrangle a system that's basically held together with duct tape and baling wire and only still works because we are the most stubborn people in the world and basically just flat out refuse to give up. I really have no words for how much we appreciate the understanding y'all have extended us over this past week.

Flat | Top-Level Comments Only

So, the issue is: for security reasons, we can't let people use certain types of HTML code in their journals (because someone malicious could use that HTML to do bad things), and a lot of the services that people want to embed from (like, say, YouTube) rely on that sort of HTML code to work. But we also don't want to not-let people post YouTube videos, that's not a solution either. So when you hit 'post' on a post that contains embed code that would otherwise be stripped from an entry and not allowed, we check it against a group of "we know this code is safe" examples, store it differently in the database, assign it a reference number, and reference it in your journal as a fake <lj-embed> tag, which uses that assigned reference number. That way, if a particular host (like YouTube) turns malicious after we 'authorize' people to embed code from that site, we can 'deauthorize' it across the whole site a lot more quickly, among many other security benefits.

The data feed from LJ, unfortunately, provides us that <lj-embed> tag in the entries feed rather than the original embed code you cut and pasted into the update window. (The reason why is a very long and boring technical digression that has to do with the original intended use of the data feed in question, but it's essentially the same security precaution.) There's no way for us to reconstruct the original embed code from just the <lj-embed> tag that's in the entry. So if, in the original entry on LJ, the security-cleaned embedded object had an ID of (say) 31337, and on LJ entry 31337 in the embed list pointed to a YouTube video of a fanvid you made, once you import it to DW, it's still pointing at the embedded object with the ID of 31337, which is a different object on DW than it was on LJ (because the embedded objects table is site-wide, not journal-specific). If embedded object ID 31337 is a YouTube video of Rick Astley singing "Never Gonna Give You Up", the fanvid you made is going to be replaced with a rickroll, because the entry-security-cleaning code that handles the embedding doesn't know that it was imported and originally pointed somewhere else.

We tried everything we possibly could think of to work around the problem, but with how LJ provides data in the data feed we use for importing, there's literally no way for us to fix it, and the only way for you to fix it is the manual repair you mention. (We also can't import polls from LJ: same problem, polls go to a separate database table that the data feed from LJ doesn't include and there's no way for us to access. For those we recommend screenshotting the poll.)

It is annoying and tedious to fix, and I'm sorry. The importer magically Just Works as much as we could make it, but that's an issue that's impossible for us to overcome.

Ah, I was wondering about that! Thanks for the explanation.

It's one of those things that is deeply annoying and also really really hard to explain because for people to 100% understand it you have to start all the way back at explaining what a CSRF attack is in the first place :/

Thank you for the explanation: it's very clear and anything more detailed would have been beyond my understanding anyway. And thanks for being so attentive to us.

One more question. If I claim my LiveJournal OpenID, as you recommend, I think it will make it a lot easier for me to add links to or fix links on the old comments of mine now on Dreamwidth that date back before the import. (What I've been doing is adding a new comment as a reply to the old one and saying, "Here's the link that belongs in the previous comment.")

My question is, does claiming my LiveJournal OpenID change anything I do on LiveJournal, or do I just log in there as I normally do and fix my old comments there if I need to without reference to OpenID, as I always have?

Edited 2022-03-17 03:44 (UTC)

It doesn't change anything you would do on the LiveJournal side.

Thank you so much for answering.

Tuesday night importer update

Re: Video embeds after the import

Re: Video embeds after the import

Re: Video embeds after the import

Re: Video embeds after the import

Re: Video embeds after the import

Re: Video embeds after the import

Re: Video embeds after the import