Tuesday night importer update

[denise]

Because computers are terrible, the thing we thought was going to be a relatively easy fix for large comment import jobs is (inevitably) turning out to be a lot more complicated, so we don't have any news for you there except "we're working on it, we think we're making progress, but we've thought that before and been wrong", sigh.

Anyone who I've told that they're on The List for us to run their import job manually: we're crunching through those; manual mode is just a lot slower than the usual automated system and the nature of the problems mean we have to stop and restart the jobs pretty frequently. I will catch up with everyone whose jobs we've managed to crunch through tomorrow or whenever we make it through the full list, whichever works out best for time management purposes! If you have a job that's failed comment imports more than 10 times, let us know the name of the account at support@dreamwidth.org and I'll put it on The List.

Thank you all very much for your patience while we work to wrangle a system that's basically held together with duct tape and baling wire and only still works because we are the most stubborn people in the world and basically just flat out refuse to give up. I really have no words for how much we appreciate the understanding y'all have extended us over this past week.

Comments

[violateraindrop]

I feel like I've seen more entries to update the users on DW in the one week I've been here than the last five years on LJ. And you actually care about your users!

Thank you for the update

[akatsija]

Thank you so much for keeping us in the loop. And it helps to know that there you are still there.

--
А.Кац и Я

[peaceflower]

Thank you for keeping us up to date on here and twitter. It doesn't involve me since I had nothing to import but I can relate to computers being a pain. Sorry it has been causing so much stress for people at DW. I hope it leads to more people staying here though and supporting DW in the future. Very grateful for all the work being done. Thank you

[erinm_4600]

♥♥♥♥

Cosmic Love

[frith]

Much love for stubborn people fighting the good fight! Rock on!

[rifleman_lincoln]

Bless you for your efforts! Keep being stubborn . . . *grin*

[rabbitica]

Thank you for everything!!!

[yourlibrarian]

Cheering you on!

Video embeds after the import

[koganbot]

Denise – Thank you for your care and stubbornness etc. Anyway, apologies if this issue was brought up recently on comment threads and I'm merely repeating it.

tl;dr version. For those of you who are considering deleting your LiveJournal once you've done your import, and your LiveJournal contains a lot of video embeds, DON'T DELETE THE LIVEJOURNAL UNTIL YOU'VE CHECKED THAT THE VIDEOS EMBEDDED CORRECTLY ON DREAMWIDTH. (Sorry for shouting; I won't do it again.) Some videos won't show up because e.g. the video is now deleted from YouTube, too bad. But others will be real, available videos that won't embed on Dreamwidth 'cause the original embed code is old or contains Shockwave Flash; sometimes Dreamwidth will indicate this with a blank space or with a message containing the word "Invalid" or "Error," but sometimes Dreamwidth will insert a different, incorrect video from somewhere else on your Dreamwidth. At least, this was my experience.

In any event, for many of these the LiveJournal embed code will be your only way of figuring what it is you originally embedded, and if you delete your LiveJournal you'll have lost this code and won't know which videos to reinsert on your Dreamwidth. You can always delete your LiveJournal later, in a couple of weeks, once you've figured out if this affects you.

Re: Video embeds after the import

[koganbot]

Longer version: I imported my lj to Dreamwidth in February 2017 and noticed that videos didn't seem to be embedding and said to myself I would deal with it later: most people looking at my old posts were still looking on lj anyway. But then a couple of years later I went to replace an old Dottie West video that YouTube had deleted, tried replacing it with another version on both platforms and saw that Dreamwidth had several wrong, not Dottie West videos. I panicked, decided I was going to insert the correct videos, did this for a number of videos I'd embedded in 2011, then life was going on and it seemed to only have an impact on 2010 and 2011, and I lost my urgency and decided to take care of it all later. Sorry – if I'd been a better citizen I'd have gotten in touch with Dreamwidth right then in 2019 and helped Dreamwidth figure out what happened.

Anyway, "later" was last week, and I discovered that every single embedded video on Dreamwidth from before February 2017 was either the wrong video or it wasn't embedded at all (with either a blank space or an error message). I've got a theory about the wrong videos: the theory might either be correct or incorrect, and I know zilch about computers, but my guess is that the Dreamwidth software saw an embed ID number that it had originally gotten from LiveJournal and was trying to match it with a current video on my Dreamwidth that had the same number. As I said, that's a guess. But it was still going on as of last week; I'd insert a correct video on a Dreamwidth entry from 2009 and an hour or day later I'd discover that video elsewhere on my Dreamwidth as well. I think I've now, since last week, fixed all the incorrect videos by manually replacing them* with the correct ones, and so far it seems that these are not getting automatically replaced by something else. I'm still trawling through the pre-import posts on Dreamwidth for the ones that say "Invalid" or "Error," and after that I'll deal with the comments where people embedded a whole lot none of which transferred to Dreamwidth but I'll be adding my own comments with links, if I get to this.

Anyway, Denise and crew are swamped so my advice is simply: if you embedded lots of videos on your LiveJournal in the past, don't delete your LiveJournal until either (1) you can correct everything yourself, or (2) the issue is somehow sorted.

*If the original video is still on YouTube but is no longer showing on LiveJournal owing to an obsolete embed code, you can click on the LiveJournal "Edit Entry" button (shows a pencil icon) and grab letters and digits from the old embed code (for example, mMo8DvW9DP8) and add them to a YouTube url after /watch?v= in place of the letters and digits that are there, and you'll see your video. And then you can hit "SHARE" and "Embed" and YouTube will give you the current embed code.

[sherlockishere]

Oh, I'm so glad to find you. So is there a list to get on for help in importing stuff from LJ over here? If so, how do I get on that list???

[denise]

I don't see that you've had any import attempts at all, much less import failures! The list is only for people who are having trouble with the automated import tool because the account they're trying to import is too big/they're getting weird errors. Start with "How do I import my journal from another site?" (and "How can I import a community from another site?" if you have a community to bring over too) and it should work without any problems unless the account has more than about 300,000 comments or in some cases if it's got data from a very, very, very long time ago. If you get any error messages in your inbox, let us know (here or at support@dreamwidth.org) and we'll help you troubleshoot.

[lj_vampirebitch]

Thank you guys so very much for all the work you're doing! I haven't been very active in either communities (lj or dw) lately but this still means the world to me to keep all my old posts and whatnot. Also, as a programmer, I do know how stressful a deadline can be, so I wish I could send over some fresh brewed coffee. Keep up the great work, there are so many people who appreciate you! Thank you!!

[denise]

Honestly, it's not even a deadline (I genuinely think it's unlikely LJ is going anywhere) so much as people being suddenly really anxious because the rumor of LJ maybe going down made them realize how vulnerable their data is and us wanting to get everything moved over for people ASAP so they could stop worrying!

And hey, another programmer! If you ever want to come join the thrill-a-minute ride that is trying to figure out why the fuck the half a million lines of 22-year-old legacy Perl is doing what it's doing, or even just fix a bunch of tiny little things here and there that are annoying you, we welcome contributions from anybody: come on over to dw_dev, and the code is on Github. (You know I have to try, heh.)

[metawidget]

Just renewed my paid account because you are stubborn and honest and generally doing the Internet right.

[denise]

Aww, thank you! That is a very good set of adjectives to have earned :)

[visionofblue]

Thank you for your hard work, and being here to save all our stuff.

[mistressofmuses]

Your stubbornness and unwillingness to give up is greatly appreciated!

[silveradept]

Stubbornness and spite are very effective methods of doing things, and here's to the importer and all those keeping it running and working, because you are doing heroic work on to of the heroic work you've been doing keeping it running even before the increased demand.

[becofoz]

Just want to say thank you again for getting my comments across. It is greatly appreciated.

[leprechaun]

I imported almost two decades worth of comments from LJ last night, successfully, on a fifth try. Thanks for fixing this!

[fredbassett]

Hi Denise, I hope life is getting easier for you folks now. A question about imports - once a big and problematic import has been done, like my own journal (looks like it was a specific problem somewhere) and primeval_denial (a metric tonne of comments) are subsequent back ups/imports likely to be less problematic?

[denise]

No, until we're able to figure out the root cause of why larger imports are dying (we keep thinking we have and then it turns out the actual issue is one layer of abstraction further down the stack) large imports will fail, sorry! We'll announce in dw-maintenance if/when we're able to figure it out.

[keplers_angels]

I just imported two LJ accts here to my existing accts here and it was so much easier than I had anticipated. Thank you!

Follow up question: I have a third journal orphaned over there on LJ ... can I import that into one my two DW accts even thought I already just ran a LJ importation into those the other day? Is there any reason why doing a a second one might cause any trouble?

[denise]

No problem at all with importing multiple journals into the same DW account.

[smutmods]

This strange, but when we sent a message to the support address you supplied we received this delivery failure a day later.. We were hoping to be put on the manual import list, but we're uncertain if that request ever made it through. Thanks for the help. Apologies if this message is redundant.

[denise]

Do you use Gmail? Gmail gets put on the block list we use occasionally when they're not fast enough in removing spammers. (It's fucking ridiculous for our block list to block Gmail, but we've conveyed this sentiment to them a few times and they continue to do it, and every time I look there isn't a good alternative that doesn't require more money than we can afford to pay, so.)

We did get your email and you are on the list. I actually think your job is running right now, so you're probably going to see things filling in over time.

[veritas_poet]

Thanks to the DW team for all your efforts and also communicating with us. I've come to use this mainly as a personal diary, since I've never been able to get much interaction or community. But I still appreciate that DW exists.