dw_maintenance | Tuesday night importer update

Tuesday night importer update

Because computers are terrible, the thing we thought was going to be a relatively easy fix for large comment import jobs is (inevitably) turning out to be a lot more complicated, so we don't have any news for you there except "we're working on it, we think we're making progress, but we've thought that before and been wrong", sigh.

Anyone who I've told that they're on The List for us to run their import job manually: we're crunching through those; manual mode is just a lot slower than the usual automated system and the nature of the problems mean we have to stop and restart the jobs pretty frequently. I will catch up with everyone whose jobs we've managed to crunch through tomorrow or whenever we make it through the full list, whichever works out best for time management purposes! If you have a job that's failed comment imports more than 10 times, let us know the name of the account at support@dreamwidth.org and I'll put it on The List.

Thank you all very much for your patience while we work to wrangle a system that's basically held together with duct tape and baling wire and only still works because we are the most stubborn people in the world and basically just flat out refuse to give up. I really have no words for how much we appreciate the understanding y'all have extended us over this past week.

Flat | Top-Level Comments Only

Denise – Thank you for your care and stubbornness etc. Anyway, apologies if this issue was brought up recently on comment threads and I'm merely repeating it.

tl;dr version. For those of you who are considering deleting your LiveJournal once you've done your import, and your LiveJournal contains a lot of video embeds, DON'T DELETE THE LIVEJOURNAL UNTIL YOU'VE CHECKED THAT THE VIDEOS EMBEDDED CORRECTLY ON DREAMWIDTH. (Sorry for shouting; I won't do it again.) Some videos won't show up because e.g. the video is now deleted from YouTube, too bad. But others will be real, available videos that won't embed on Dreamwidth 'cause the original embed code is old or contains Shockwave Flash; sometimes Dreamwidth will indicate this with a blank space or with a message containing the word "Invalid" or "Error," but sometimes Dreamwidth will insert a different, incorrect video from somewhere else on your Dreamwidth. At least, this was my experience.

In any event, for many of these the LiveJournal embed code will be your only way of figuring what it is you originally embedded, and if you delete your LiveJournal you'll have lost this code and won't know which videos to reinsert on your Dreamwidth. You can always delete your LiveJournal later, in a couple of weeks, once you've figured out if this affects you.

Longer version: I imported my lj to Dreamwidth in February 2017 and noticed that videos didn't seem to be embedding and said to myself I would deal with it later: most people looking at my old posts were still looking on lj anyway. But then a couple of years later I went to replace an old Dottie West video that YouTube had deleted, tried replacing it with another version on both platforms and saw that Dreamwidth had several wrong, not Dottie West videos. I panicked, decided I was going to insert the correct videos, did this for a number of videos I'd embedded in 2011, then life was going on and it seemed to only have an impact on 2010 and 2011, and I lost my urgency and decided to take care of it all later. Sorry – if I'd been a better citizen I'd have gotten in touch with Dreamwidth right then in 2019 and helped Dreamwidth figure out what happened.

Anyway, "later" was last week, and I discovered that every single embedded video on Dreamwidth from before February 2017 was either the wrong video or it wasn't embedded at all (with either a blank space or an error message). I've got a theory about the wrong videos: the theory might either be correct or incorrect, and I know zilch about computers, but my guess is that the Dreamwidth software saw an embed ID number that it had originally gotten from LiveJournal and was trying to match it with a current video on my Dreamwidth that had the same number. As I said, that's a guess. But it was still going on as of last week; I'd insert a correct video on a Dreamwidth entry from 2009 and an hour or day later I'd discover that video elsewhere on my Dreamwidth as well. I think I've now, since last week, fixed all the incorrect videos by manually replacing them* with the correct ones, and so far it seems that these are not getting automatically replaced by something else. I'm still trawling through the pre-import posts on Dreamwidth for the ones that say "Invalid" or "Error," and after that I'll deal with the comments where people embedded a whole lot none of which transferred to Dreamwidth but I'll be adding my own comments with links, if I get to this.

Anyway, Denise and crew are swamped so my advice is simply: if you embedded lots of videos on your LiveJournal in the past, don't delete your LiveJournal until either (1) you can correct everything yourself, or (2) the issue is somehow sorted.

*If the original video is still on YouTube but is no longer showing on LiveJournal owing to an obsolete embed code, you can click on the LiveJournal "Edit Entry" button (shows a pencil icon) and grab letters and digits from the old embed code (for example, mMo8DvW9DP8) and add them to a YouTube url after /watch?v= in place of the letters and digits that are there, and you'll see your video. And then you can hit "SHARE" and "Embed" and YouTube will give you the current embed code.

Edited 2022-03-16 19:35 (UTC)

So, the issue is: for security reasons, we can't let people use certain types of HTML code in their journals (because someone malicious could use that HTML to do bad things), and a lot of the services that people want to embed from (like, say, YouTube) rely on that sort of HTML code to work. But we also don't want to not-let people post YouTube videos, that's not a solution either. So when you hit 'post' on a post that contains embed code that would otherwise be stripped from an entry and not allowed, we check it against a group of "we know this code is safe" examples, store it differently in the database, assign it a reference number, and reference it in your journal as a fake <lj-embed> tag, which uses that assigned reference number. That way, if a particular host (like YouTube) turns malicious after we 'authorize' people to embed code from that site, we can 'deauthorize' it across the whole site a lot more quickly, among many other security benefits.

The data feed from LJ, unfortunately, provides us that <lj-embed> tag in the entries feed rather than the original embed code you cut and pasted into the update window. (The reason why is a very long and boring technical digression that has to do with the original intended use of the data feed in question, but it's essentially the same security precaution.) There's no way for us to reconstruct the original embed code from just the <lj-embed> tag that's in the entry. So if, in the original entry on LJ, the security-cleaned embedded object had an ID of (say) 31337, and on LJ entry 31337 in the embed list pointed to a YouTube video of a fanvid you made, once you import it to DW, it's still pointing at the embedded object with the ID of 31337, which is a different object on DW than it was on LJ (because the embedded objects table is site-wide, not journal-specific). If embedded object ID 31337 is a YouTube video of Rick Astley singing "Never Gonna Give You Up", the fanvid you made is going to be replaced with a rickroll, because the entry-security-cleaning code that handles the embedding doesn't know that it was imported and originally pointed somewhere else.

We tried everything we possibly could think of to work around the problem, but with how LJ provides data in the data feed we use for importing, there's literally no way for us to fix it, and the only way for you to fix it is the manual repair you mention. (We also can't import polls from LJ: same problem, polls go to a separate database table that the data feed from LJ doesn't include and there's no way for us to access. For those we recommend screenshotting the poll.)

It is annoying and tedious to fix, and I'm sorry. The importer magically Just Works as much as we could make it, but that's an issue that's impossible for us to overcome.

Ah, I was wondering about that! Thanks for the explanation.

It's one of those things that is deeply annoying and also really really hard to explain because for people to 100% understand it you have to start all the way back at explaining what a CSRF attack is in the first place :/

Thank you for the explanation: it's very clear and anything more detailed would have been beyond my understanding anyway. And thanks for being so attentive to us.

One more question. If I claim my LiveJournal OpenID, as you recommend, I think it will make it a lot easier for me to add links to or fix links on the old comments of mine now on Dreamwidth that date back before the import. (What I've been doing is adding a new comment as a reply to the old one and saying, "Here's the link that belongs in the previous comment.")

My question is, does claiming my LiveJournal OpenID change anything I do on LiveJournal, or do I just log in there as I normally do and fix my old comments there if I need to without reference to OpenID, as I always have?

Edited 2022-03-17 03:44 (UTC)

It doesn't change anything you would do on the LiveJournal side.

Thank you so much for answering.

Yes, the embeds are rendered in the entry as, iirc, something along the lines of "embed # xxxx" (except in Dreamwidth pseudo-html). Excellent pattern-matching!

Flat | Top-Level Comments Only

Tuesday night importer update

Video embeds after the import

Re: Video embeds after the import

Re: Video embeds after the import

Re: Video embeds after the import

Re: Video embeds after the import

Re: Video embeds after the import

Re: Video embeds after the import

Re: Video embeds after the import

Re: Video embeds after the import

Re: Video embeds after the import