dw_maintenance | The importer is still running, we swear! It's just very busy.

The importer is still running, we swear! It's just very busy.

People who are worried because the import of their LiveJournal account has been running for a long time with no real signs of progress: please don't be concerned! The import queue is currently a little long. (In the same way that Mt Everest is a little tall and the Mariana Trench is a little deep.) We're limited in how many import jobs we can run simultaneously and how quickly we can start the next import after one finishes: LiveJournal, like all sites, has restrictions on how frequently we can programmatically request data from their site, so the import queue can get very backed up at times like this when more imports are being started than are finishing. If you look at the import queue and the numbers don't seem to be changing much, or are only going up, it doesn't mean that no imports are finishing: it means a lot of additional people have scheduled an import since the last time you reloaded.

As long as you haven't gotten a failure message in your on-site inbox, your import is still running. (Even if you have gotten a failure message, your import may still be running: if the site thinks that the failure is something that might correct itself, like being unable to connect to the remote site, it will retry for a few times before giving up.) If you have gotten a failure message, the error message in your inbox should tell you what went wrong.

The three common problems right now: 1) you mistyped your username and/or password; 2) you need to agree to LiveJournal's new ToS before they'll permit you to access the data in your account; 3) an entry or entries in your LiveJournal account have a text encoding mismatch and you need to follow the link in the error message to fix it on LJ.

If you haven't gotten a failure message, your import is still waiting in the queue, and will run when it makes its way up to the top of the queue.

People keep asking us how long the queue is (by which they mean, how much time will it take for a job just started to successfully finish: length of time, not number of jobs waiting). I would love to be able to give you a definite answer! It's really, really hard for us to predict how long it will take for a job to get up to the top of the queue, though: how long an import takes to complete depends on a lot of things, including how many posts/comments are in the journal. To give you a ballpark figure that might be off by up to 100% on either side: If I personally started a brand new import right now (in my timezone, the early morning of Monday 10 Apr), I would be pleasantly surprised if it finished before Tuesday morning (24 hours or so), would expect it to finish sometime on Tuesday night or maybe even stretch all the way to Wednesday night (36-60 hours), and wouldn't start to wonder if I should poke

mark or

alierak to doublecheck that something hadn't gotten stuck in such a way that our monitoring didn't alert us about it until Thursday afternoon or evening (100+ hours).

All of those time estimates, by the way, assume a relatively uncomplicated job that succeeds on the first try. When the site tries again after a failure, it includes a delay that increases after each failure in case the failure was due to transient network issues. So, I know some of you started an import at the end of last week and it's still running: some of you are trying to import very large journals, and some of you ran into errors along the way and are in a retry wait loop. Again: if you haven't gotten the final error message in your inbox (and it will tell you it's the final error), it's still chugging along.

You do not have to leave the importer page open or stay logged into Dreamwidth until your import finishes. (You do have to avoid changing your LJ password until the job is done, or it will fail.) You can close the window/tab and go off and explore Dreamwidth; the movers will be along in a little while with your stuff.

The tl;dr version of my usual longwinded babble: IMPORTER VERY BUSY. MANY PEOPLE MOVING IN. LIKE ON DORM OR APARTMENT MOVE-IN DAY, FREIGHT ELEVATORS VERY SLOW. BUILDING OWNERS RUNNING FREIGHT ELEVATORS AS FAST AS POSSIBLE AND APOLOGIZE FOR THE WAIT.

A housewarming glass of champagne/sparkling cider/fancy handmade soda for all! Welcome to the neighborhood.

EDIT, 10 Apr 2017 7:15PM EDT: the importer is not the only thing that is very busy today! I'm trying to get to all the comments here, but keep getting dragged off to handle other stuff (and will be knocking off for the night soon). If you have a technical support problem, it will probably be faster to open a support request, where there's less likelihood that it will get overlooked in the sea of comments.

Threaded | Top-Level Comments Only

Hmm. Interesting - so I can import just comments. Does that work even if the people commenting don't have DW accounts? *should just wander off to check the FAQ*

Thanks for the information!

I hope that by now you're subsisting on more than calorie free soda! Thank you for all the hard work you're putting in with this new influx.

Heh, I don't mean to distract from the very real saudade we're all feeling, but I do have to say that holy fuck am I glad LJ didn't exist when I was a teen! Having my early 20s preserved forever is bad enough. ;) (I was just on the edge of the digital-childhood generation; my peers and I didn't get "proper" internet access until college for the most part and our teen years were spent on BBSes that were ephemeral enough that my teenage bad life choices are not immortlized forever. Thank God.)

Ahem.

All joking aside, though, I know exactly what people are feeling, or at least to an extent; I stuck it out with working for Six Apart (owners at the time I left) long past the point where I should have given up for my health and my sanity, because of how much I loved LJ, and even though by the time I quit I was swearing I would never work on anything even in the same neighborhood as social media ever again, it was eight months later when

mark and I started having the conversations that turned into DW. But no matter how amazing it is that people were willing to take a gamble on us in the beginning and have proven our wild-eyed idealism would actually work when put into practice, and no matter how glad I am that DW exists and can give people that escape hatch/lifeboat of saving their digital history even if they don't decide to shift entirely to DW from LJ/Facebook/Tumblr/etc, it's still incredibly sad to watch something we all poured so much of ourselves into start to crumble like this, isn't it?

I mean, DW isn't LJ, and will never be LJ (and we've never even tried to be LJ; we're us, not them, even if "them" used to be our younger selves once upon a time). So even if the vast majority of people who've ever used LJ did move their stuff over, which is unlikely, it's never going to be the same thing. (And that's not even getting into how many of my friends have died in the past 15 years and whose journals may get closed at any point from here on out, now, according to the new LJ ToS; I'm trying really hard not to think about that!) It's understandable that people are sad about it. Hell, I'm sad about it!

Shoot, that is a fantastic suggestion. I'mma submit it to

dreamwidth-help.

You can! You do have to import your entries along with the comments, though, or else there's nothing to stick them on. :)

Imported comments are displayed as coming from "so-and-so.livejournal.com" and, thanks to the magic of OpenID, as long as you still have access to the LJ account you were logged into when you originally made the comment on LJ, you can log into DW using that OpenID account and still have the same level of control (edit, delete, etc) as you had for the LJ version.

Or, you can claim your OpenID with your DW account, and then all the comments you've left in other people's journals with that LJ account will update to display as your DW account instead. For instance, look at the comments to this post from 2008, which was originally made on LJ, and was the first time that we talked about DW in public. I imported the entry, and all its comments, from LJ; the first comment thread, now appearing to be from me and zorkian (which is mark's personal account like synecdochic is my personal account), originally displayed like how many of the others still do. (As xb95.livejournal.com and synecdochic.livejournal.com, respectively.) Then I claimed my synecdochic.livejournal.com OpenID URL with synecdochic, and he claimed his xb95.livejournal.com OpenID URL with zorkian, and now anywhere on the site that somebody has imported a comment that I made while logged into LJ as synecdochic, it shows as synecdochic instead!

(It's even smart enough that if you have the same icon keywords on DW as you did on LJ, after you claim your OpenID, all the comments will even display the proper icon.)

I imported my journal a long time ago, and my community entries did not come with. I'm pretty sure they won't be coming with for the latest importees, either.

I'm fairly certain that, in order to get those entries over to Dreamwidth without manually re-posting them, you have to import the actual community. Which is an option! If you're one of the community moderators on LJ, you can probably do it yourself even.

Good luck!

AHA, I misunderstood - glad to understand this better!

Hmmm, so. I ported my journal over when DW was born (I'm an early user!) and I've been cross-posting since then using Semagic. However, I've a load of comments on LJ (when it was more active) than I do here, because see: more active.

If I imported those comments, they'd come with the posts to which they're attached, which would double my posts, wouldn't it?

I'm not going to do anything with this right now, as the import system is understandably overwhelmed, but it's something to think about. It would be nice to keep those comments, if at all possible.

That is indeed Very Weird on first glance, but I did some quick spot-checking of a few of the entries that are dated 2015, and I think I've figured it out! It turns out that all of the ones I checked are entries that were actually posted to Dreamwidth today. (I'm guessing that the owners manually imported from LJ using copy/paste, instead of using our on-site import tool: entries that are imported through our import tool don't appear on reading pages, to prevent someone from importing a huge journal and flooding your reading page completely.) In each case, the poster manually adjusted the time on the timestamp to be from 2015 while they were writing the entry, so that the entries sorted themselves into the right place in their archives: the contents of a journal show on the recent entries/archive/etc pages based on the user-supplied time, so you can belatedly write that review of the awesome concert you went to in 2001 (and just found your notes for) and it will happily fit itself in along the rest of your entries from 2001.

However! Because timezones are a thing, and trying to convert a user-supplied time into a time that can be manipulated to account for varying timezones is the sort of problem that looks easy and is actually hard enough that it often makes experienced programmers weep wretchedly into their keyboards, and if we did it the other way a misconfigured computer clock could mean nobody would ever see that entry you just posted, the reading page (as opposed to the recent entries/archive/year/month/day/etc pages) is ordered by "what time did our servers receive this post", not "what time did the user put on the timestamp": we completely ignore the timestamp and just go by the actual order in which they got to us. So, if I write that belated review of that Moxy Früvous show I went to in 2001 and forgot to document, it will show in my journal in between the other reviews of the other Moxy Früvous shows I went to in 2001 (I, uh, went to a lot of Moxy Früvous shows in 1999-2001), but you'll see it on your reading page in between things the other people you're subscribed to have posted that are dated 10 April 2017.

Does that explanation make enough sense to you? (I've been awake and working way too long today; I'm losing the ability to tell if I'm writing coherently!) In essence, it is a glitch related to the mass migration and will fix itself, in that people who are manually importing and changing the dates on their entries will get all of everything over here sooner or later.

I am literally sitting down to dinner with my lovely wife as I type this!

That does make sense, except that whatever is borked, it screwed up the order of posts that were actually made recently by other people on my reading list. So much so that I have 3 posts from yesterday -- not from folks I just added; we've followed each other for a while -- in between two from today. And all today's posts from people I have been following a while are knocked out of order, so they go from like 10 a.m. to 5 a.m. to 8 to 5:55 p.m. to 4:35. That sort of thing.

I am baffled. I'll wait and see if it fixes itself and get back to you.

It's nice to see people coming to DW. Hopefully you'll have wider audience and more helpers - devs and/or translators. Just don't sell us to SUP. I've been blogging on LJ since 2004, and it was the worst thing that happened to our old platform.

Thanks for giving us an alternative to LJ, etc. I've had a free account for a while and just imported my LJ to DW. My import went perfectly if a little long (no biggie for me).

I still am getting updates in my inbox, though. It's the same three messages over and over. Not sure why this is happening but wanted to let you know it is.

To echo a couple of other comments, I just wanted to say thank you for being a great alternative to LJ! And for making it so easy to transfer everything.

Here's hoping more people from certain communities make the move...

Even without the recent drama, LJ technical support has always been shit lol.

Hmmmm. Okay, I've got one more theory (and if I'm right, the answer is "don't worry about it, it will fix itself in a bit") and it comes with two questions:

1) Did it, by any chance, just fix itself? (Or rather, did the thing I just did fix it?)

2) If not: can you give me the usernames of the people whose posts from yesterday are misbehaving, and the usernames of the people who posted the two from today? (Like, in such a way that I can tell which group is which, I mean.) I don't need the URLs of the entries, just the usernames. I Have A Theory.

Thank you! We pretty much operate on the theory of "if it's something we would be curious about/would want to know if we were a user on a site like this, we'll happily talk your ear off about it". :) (Okay, some of that is because I don't think I could be terse if you paid me a million dollars and I weep when someone tries to take away my parentheticals, but still.)

We're a super tiny company, so we can't offer the same features that a lot of other sites have. None of us are actually able to work on the site for the 40-hours-a-week that traditional business America considers "full time", much less the utterly horrifying 60-80 hours a week that the tech industry considers mandatory (I'm disabled and my work availability depends on how stupid my body and brain are being; Mark has a dayjob and a kid; etc), so our development/improvement pace and the rate at which we're able to do features and bugfixes is never going to match what, say, Facebook or Tumblr can accomplish with their hundreds of software engineers and designers. We're never going to be the next big thing or get a spot in the 100 most visited sites on the internet (and we don't even really that to happen in the first place!)

But we damn well can promise you that we'll always explain why we're doing things or changing things, we'll never accept advertising (so we'll never alter our acceptable-content policies or features based on what advertisers want), and we'll never take venture capital (so we're the only people who have the right to make decisions about what happens to the site and we can look at the (very) long term good of the service instead of having to chase the super-huge return on investment to recoup the VC's money). And enough people seem to think that's a good enough trade that we've been able to keep things running this far. :)

Speaking for myself, I thank you for coming out of hiding. Especially for this. No reply needed.

Edited 2017-04-10 23:53 (UTC)

Yeah... That wasn't an option back in 2010 or so, when I deleted my LJ and I so, so wish it had been! :(

The metaphor works so well for far too many of us, doesn't it?

Aww, thanks! Yeah, I'm glad I did too, even though I really needed those eight months of mental health break at the time...

When you import, cross posts originally made on DW don't duplicate. I had comments on both sites and would import my comments every 3 to 6 months in order to have then all in the same place.

Nope, still borked. Out-of-order entries are all those which, as I scroll down my reading page, fall below the ones by lunadelcorvo, who is the person with years-old posts showing as new.

So they are posts by:

zulu (2:11 pm)
ysabetwordsmith (2:47 pm)
ysabetwordsmith (1:24 pm)
dianeduane (5:55 pm)
ranunculus (7:54 am)
deelaundry (10:19 am)
dw_maintenance (5:53 am)
jadesfire (8:26 am)

All of the above are dated today, April 10, and are way out of their proper chronological order. Then there are the April 9 entries:

ranunculus
ysabetwordsmith
topaz_eyes

And then we're back to today, but still out of chronological order:

taiga13 (7:54 am)

Then more of yesterday's posts from ysabet mostly.

I'm guessing she's one of a few people who've just imported entries they made in the last day or so, so they're showing up out of order. But entries by dw_maintenance, deelaundry, topaz_eyes, zulu, jadesfire -- I have no idea what happened there. In any case it only appears to be affecting the first two pages of my reading list, so it's probably a one-time thing and I'm not worried about it.

In addition to

sporky_rat's helpful links, I want to specifically mention that unlike LJ, where permanent accounts had higher limits than paid accounts, on DW a seed account has exactly the same capabilities and limits and features as a premium paid account. (We have two levels of paid service, paid and premium paid; premium paid time costs more, but gives you higher limits/quotas/capabilities/etc.) So there isn't a special account type that gets extra stuff that people don't usually have access to; literally the only benefit of a seed account over a premium paid account is that you don't have to keep paying us money for it. (Which, yes, is a big benefit! But it's not like seed accounts get double the icons of premium paid accounts or anything.) We did it that way because we knew we were planning on only selling them once, and we didn't want to set up a situation where people could see other people using benefits it wasn't possible to still get.

(We did wind up having to go back on that 'once only' plan: in 2010, we had a long stretch of not being able to accept payments, because PayPal demanded that we censor a bunch of our users' writing that was perfectly legal but PayPal didn't like because it talked about human beings! having sex! We told them "no we won't", they said "then we won't process payments for you anymore, and by the way, your account is frozen", we said a lot of rude words, a lot of other stuff happened, and it was three months before we could take payments again. During that time, we kept the site running with the money we had in reserve from that first seed account sale, and once we could take payments again, we sold another batch of seed accounts to replenish the emergency fund. But we really hope we're never going to have to do it again, for all the reasons I went over in that post from before we launched in 2009.)

Basically, unlike LJ where "permanent account" is a whole separate status with different levels of service, here on DW it's more like "thank you for taking a chance and having faith in us when nobody could be sure we'd last long enough that you'd get all the four years of service your $200 would have paid for if you paid for it separately". In exchange for having provided our seed funding (hence the name 'seed account'), the "interest" that we've repaid them is not having to pay us again. :)

I wonder if that works for all crossposts - I use a third party program called Semagic.

I'll do some testing, when things are less chaotic.

Thank you!

YES I AM BEING A PEST I'm sorry.

But I just realized none of my "memories" links imported with the rest of my LJ.

Is there an easy way to get those over, or is it one of those things that LJ is stubborn about and we'll just have to do it by hand?

And also: THANK YOU. Seriously, thank you so much for everything you do. The DW staff is simply wonderful.

Edited 2017-04-11 00:24 (UTC)

Threaded | Top-Level Comments Only

Page 9 of 19

The importer is still running, we swear! It's just very busy.

no subject

no subject

no subject

no subject

no subject

no subject

misleading stats (newbyday)

no subject

no subject

no subject

no subject

no subject

no subject

Thank you!

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

Question about importing "memories" links