The importer is still running, we swear! It's just very busy.

[denise]

People who are worried because the import of their LiveJournal account has been running for a long time with no real signs of progress: please don't be concerned! The import queue is currently a little long. (In the same way that Mt Everest is a little tall and the Mariana Trench is a little deep.) We're limited in how many import jobs we can run simultaneously and how quickly we can start the next import after one finishes: LiveJournal, like all sites, has restrictions on how frequently we can programmatically request data from their site, so the import queue can get very backed up at times like this when more imports are being started than are finishing. If you look at the import queue and the numbers don't seem to be changing much, or are only going up, it doesn't mean that no imports are finishing: it means a lot of additional people have scheduled an import since the last time you reloaded.

As long as you haven't gotten a failure message in your on-site inbox, your import is still running. (Even if you have gotten a failure message, your import may still be running: if the site thinks that the failure is something that might correct itself, like being unable to connect to the remote site, it will retry for a few times before giving up.) If you have gotten a failure message, the error message in your inbox should tell you what went wrong.

The three common problems right now: 1) you mistyped your username and/or password; 2) you need to agree to LiveJournal's new ToS before they'll permit you to access the data in your account; 3) an entry or entries in your LiveJournal account have a text encoding mismatch and you need to follow the link in the error message to fix it on LJ.

If you haven't gotten a failure message, your import is still waiting in the queue, and will run when it makes its way up to the top of the queue.

People keep asking us how long the queue is (by which they mean, how much time will it take for a job just started to successfully finish: length of time, not number of jobs waiting). I would love to be able to give you a definite answer! It's really, really hard for us to predict how long it will take for a job to get up to the top of the queue, though: how long an import takes to complete depends on a lot of things, including how many posts/comments are in the journal. To give you a ballpark figure that might be off by up to 100% on either side: If I personally started a brand new import right now (in my timezone, the early morning of Monday 10 Apr), I would be pleasantly surprised if it finished before Tuesday morning (24 hours or so), would expect it to finish sometime on Tuesday night or maybe even stretch all the way to Wednesday night (36-60 hours), and wouldn't start to wonder if I should poke mark or alierak to doublecheck that something hadn't gotten stuck in such a way that our monitoring didn't alert us about it until Thursday afternoon or evening (100+ hours).

All of those time estimates, by the way, assume a relatively uncomplicated job that succeeds on the first try. When the site tries again after a failure, it includes a delay that increases after each failure in case the failure was due to transient network issues. So, I know some of you started an import at the end of last week and it's still running: some of you are trying to import very large journals, and some of you ran into errors along the way and are in a retry wait loop. Again: if you haven't gotten the final error message in your inbox (and it will tell you it's the final error), it's still chugging along.

You do not have to leave the importer page open or stay logged into Dreamwidth until your import finishes. (You do have to avoid changing your LJ password until the job is done, or it will fail.) You can close the window/tab and go off and explore Dreamwidth; the movers will be along in a little while with your stuff.

The tl;dr version of my usual longwinded babble: IMPORTER VERY BUSY. MANY PEOPLE MOVING IN. LIKE ON DORM OR APARTMENT MOVE-IN DAY, FREIGHT ELEVATORS VERY SLOW. BUILDING OWNERS RUNNING FREIGHT ELEVATORS AS FAST AS POSSIBLE AND APOLOGIZE FOR THE WAIT.

A housewarming glass of champagne/sparkling cider/fancy handmade soda for all! Welcome to the neighborhood.

EDIT, 10 Apr 2017 7:15PM EDT: the importer is not the only thing that is very busy today! I'm trying to get to all the comments here, but keep getting dragged off to handle other stuff (and will be knocking off for the night soon). If you have a technical support problem, it will probably be faster to open a support request, where there's less likelihood that it will get overlooked in the sea of comments.

Comments

[blackmare]

Okay, so I just went into my "Circles" settings and added several folks to my reading list who were already friends but who for whatever reason I completely forgot to follow.

And now my Reading page is TOTALLY OUT OF ORDER chronologically. Like, the post showing at the top is from June 2015. And even when I get to the posts from today, they're not in order with newest at the top anymore. They're just ... haphazard.

What on earth happened? I didn't change any other settings, just added some people to my Reading list.

EDIT: in case it's not clear, I'm not angry about this, just bemused and wondering if or how I can fix it, or whether it's a glitch related to the mass migration and will fix itself. Indeed, I am now seeing a couple of today's posts above the ones from 2015, and things seem to be rearranging on their own somewhat.

[denise]

That is indeed Very Weird on first glance, but I did some quick spot-checking of a few of the entries that are dated 2015, and I think I've figured it out! It turns out that all of the ones I checked are entries that were actually posted to Dreamwidth today. (I'm guessing that the owners manually imported from LJ using copy/paste, instead of using our on-site import tool: entries that are imported through our import tool don't appear on reading pages, to prevent someone from importing a huge journal and flooding your reading page completely.) In each case, the poster manually adjusted the time on the timestamp to be from 2015 while they were writing the entry, so that the entries sorted themselves into the right place in their archives: the contents of a journal show on the recent entries/archive/etc pages based on the user-supplied time, so you can belatedly write that review of the awesome concert you went to in 2001 (and just found your notes for) and it will happily fit itself in along the rest of your entries from 2001.

However! Because timezones are a thing, and trying to convert a user-supplied time into a time that can be manipulated to account for varying timezones is the sort of problem that looks easy and is actually hard enough that it often makes experienced programmers weep wretchedly into their keyboards, and if we did it the other way a misconfigured computer clock could mean nobody would ever see that entry you just posted, the reading page (as opposed to the recent entries/archive/year/month/day/etc pages) is ordered by "what time did our servers receive this post", not "what time did the user put on the timestamp": we completely ignore the timestamp and just go by the actual order in which they got to us. So, if I write that belated review of that Moxy Früvous show I went to in 2001 and forgot to document, it will show in my journal in between the other reviews of the other Moxy Früvous shows I went to in 2001 (I, uh, went to a lot of Moxy Früvous shows in 1999-2001), but you'll see it on your reading page in between things the other people you're subscribed to have posted that are dated 10 April 2017.

Does that explanation make enough sense to you? (I've been awake and working way too long today; I'm losing the ability to tell if I'm writing coherently!) In essence, it is a glitch related to the mass migration and will fix itself, in that people who are manually importing and changing the dates on their entries will get all of everything over here sooner or later.

[blackmare]

That does make sense, except that whatever is borked, it screwed up the order of posts that were actually made recently by other people on my reading list. So much so that I have 3 posts from yesterday -- not from folks I just added; we've followed each other for a while -- in between two from today. And all today's posts from people I have been following a while are knocked out of order, so they go from like 10 a.m. to 5 a.m. to 8 to 5:55 p.m. to 4:35. That sort of thing.

I am baffled. I'll wait and see if it fixes itself and get back to you.

[denise]

Hmmmm. Okay, I've got one more theory (and if I'm right, the answer is "don't worry about it, it will fix itself in a bit") and it comes with two questions:

1) Did it, by any chance, just fix itself? (Or rather, did the thing I just did fix it?)

2) If not: can you give me the usernames of the people whose posts from yesterday are misbehaving, and the usernames of the people who posted the two from today? (Like, in such a way that I can tell which group is which, I mean.) I don't need the URLs of the entries, just the usernames. I Have A Theory.

[blackmare]

Nope, still borked. Out-of-order entries are all those which, as I scroll down my reading page, fall below the ones by lunadelcorvo, who is the person with years-old posts showing as new.

So they are posts by:

zulu (2:11 pm)
ysabetwordsmith (2:47 pm)
ysabetwordsmith (1:24 pm)
dianeduane (5:55 pm)
ranunculus (7:54 am)
deelaundry (10:19 am)
dw_maintenance (5:53 am)
jadesfire (8:26 am)

All of the above are dated today, April 10, and are way out of their proper chronological order. Then there are the April 9 entries:

ranunculus
ysabetwordsmith
topaz_eyes

And then we're back to today, but still out of chronological order:

taiga13 (7:54 am)

Then more of yesterday's posts from ysabet mostly.

I'm guessing she's one of a few people who've just imported entries they made in the last day or so, so they're showing up out of order. But entries by dw_maintenance, deelaundry, topaz_eyes, zulu, jadesfire -- I have no idea what happened there. In any case it only appears to be affecting the first two pages of my reading list, so it's probably a one-time thing and I'm not worried about it.

[denise]

Okay, nevermind, that knocks out the theory I was theorizing!

Yeah, at this point I could start a huge long investigation, or I could fall back on "...sometimes people using the importer can make your reading page a bit weird for a while; if it's still happening in a week, let us know". For the sake of getting to bed sometime this century, I think I'm gonna go with "if it's still happening in a week, let us know". :)

(The code that actually assembles and organizes your reading page has been around for a very long time; it's usually pretty stable, but sometimes it decides that it's going to do something TOTALLY UNEXPECTED for a little bit to keep us on our toes. I like to think of it as: it's actually eighteen years old by now (no shit), so it's trying out its independence.)

[blackmare]

Hey, I appreciate you even taking a quick look at it; this was way above and beyond the call of duty. Rest well and don't sweat it, and thank you again.