The importer is still running, we swear! It's just very busy.

[denise]

People who are worried because the import of their LiveJournal account has been running for a long time with no real signs of progress: please don't be concerned! The import queue is currently a little long. (In the same way that Mt Everest is a little tall and the Mariana Trench is a little deep.) We're limited in how many import jobs we can run simultaneously and how quickly we can start the next import after one finishes: LiveJournal, like all sites, has restrictions on how frequently we can programmatically request data from their site, so the import queue can get very backed up at times like this when more imports are being started than are finishing. If you look at the import queue and the numbers don't seem to be changing much, or are only going up, it doesn't mean that no imports are finishing: it means a lot of additional people have scheduled an import since the last time you reloaded.

As long as you haven't gotten a failure message in your on-site inbox, your import is still running. (Even if you have gotten a failure message, your import may still be running: if the site thinks that the failure is something that might correct itself, like being unable to connect to the remote site, it will retry for a few times before giving up.) If you have gotten a failure message, the error message in your inbox should tell you what went wrong.

The three common problems right now: 1) you mistyped your username and/or password; 2) you need to agree to LiveJournal's new ToS before they'll permit you to access the data in your account; 3) an entry or entries in your LiveJournal account have a text encoding mismatch and you need to follow the link in the error message to fix it on LJ.

If you haven't gotten a failure message, your import is still waiting in the queue, and will run when it makes its way up to the top of the queue.

People keep asking us how long the queue is (by which they mean, how much time will it take for a job just started to successfully finish: length of time, not number of jobs waiting). I would love to be able to give you a definite answer! It's really, really hard for us to predict how long it will take for a job to get up to the top of the queue, though: how long an import takes to complete depends on a lot of things, including how many posts/comments are in the journal. To give you a ballpark figure that might be off by up to 100% on either side: If I personally started a brand new import right now (in my timezone, the early morning of Monday 10 Apr), I would be pleasantly surprised if it finished before Tuesday morning (24 hours or so), would expect it to finish sometime on Tuesday night or maybe even stretch all the way to Wednesday night (36-60 hours), and wouldn't start to wonder if I should poke mark or alierak to doublecheck that something hadn't gotten stuck in such a way that our monitoring didn't alert us about it until Thursday afternoon or evening (100+ hours).

All of those time estimates, by the way, assume a relatively uncomplicated job that succeeds on the first try. When the site tries again after a failure, it includes a delay that increases after each failure in case the failure was due to transient network issues. So, I know some of you started an import at the end of last week and it's still running: some of you are trying to import very large journals, and some of you ran into errors along the way and are in a retry wait loop. Again: if you haven't gotten the final error message in your inbox (and it will tell you it's the final error), it's still chugging along.

You do not have to leave the importer page open or stay logged into Dreamwidth until your import finishes. (You do have to avoid changing your LJ password until the job is done, or it will fail.) You can close the window/tab and go off and explore Dreamwidth; the movers will be along in a little while with your stuff.

The tl;dr version of my usual longwinded babble: IMPORTER VERY BUSY. MANY PEOPLE MOVING IN. LIKE ON DORM OR APARTMENT MOVE-IN DAY, FREIGHT ELEVATORS VERY SLOW. BUILDING OWNERS RUNNING FREIGHT ELEVATORS AS FAST AS POSSIBLE AND APOLOGIZE FOR THE WAIT.

A housewarming glass of champagne/sparkling cider/fancy handmade soda for all! Welcome to the neighborhood.

EDIT, 10 Apr 2017 7:15PM EDT: the importer is not the only thing that is very busy today! I'm trying to get to all the comments here, but keep getting dragged off to handle other stuff (and will be knocking off for the night soon). If you have a technical support problem, it will probably be faster to open a support request, where there's less likelihood that it will get overlooked in the sea of comments.

Comments

Re: Kind and polite

[nodrog]

They've always been this way.  - No, that's not brown-nosing; the staffers are all ex-LJ, and determined to avoid the mistakes of their predecessor.  Including that one.

[aflon]

Thank you for being an ethical haven :)

I've had a Journal here since 2009.

Just waiting for comments to be imported and then it'll be time to kick LJ to the kerb.

[denise]

To be scrupulously fair because that's how I try to roll, it doesn't necessarily have to be actively evil, just the law of unintended consequence. The way LJ (and DW, since we're a code fork of LJ) is programmed, the best way to keep someone from posting or commenting while logged in until they agree to a new version of the ToS just so happens to also fire when you're using the XML-RPC protocol to download journal contents, and DW's importer uses the XML-RPC protocol. That "you can't do anything until you agree to the new ToS version" was actually added ages ago. (I want to say ...2005? 2006? Something like that. While I was still working for LJ and it was still owned by Six Apart, at least.) So, it's possible the current team went to go build a ToS check, realized there was support for doing one already in the code, and enabled it, without specifically realizing/actively deciding that it would also block people from accessing their data through the XML-RPC protocol.

Mind you, I don't know for sure one way or the other! I just like to point out when the fail is possibly accidental instead of having to automatically be malicious. :)

[denise]

Glad it was nice and zippy for you! And welcome home.

[denise]

MAKE MINE A NICE DRAGON AGE EPIC

[kareila]

No problem - post away!

[burbilog]

In practice, we sometimes need to restrict accessing the site via Tor, because much of the malicious traffic we get comes through Tor: not all Tor traffic is malicious, of course, but 99% of our malicious traffic comes via Tor and sometimes we have to implement restrictions in order to stop it.

May be it's better to create dreamwidthXXXXX.onion and handle Tor traffic as a native Tor service?

[elfie_chan]

You are awesome, DW Crew. I transferred my LJ over here ages ago, but I still used to go and check into LJ to read my friends page. No longer. Thanks for being here.

[princess_kessie]

And I just got mine completed after three days :D

Thanks for all you do, DW team xx

[denise]

Glad to have you! And we certainly do try to have fun, although sometimes I should have my laptop taken away from me until my ADHD meds stabilize. ;)

[denise]

It's been an interesting week!

[mlknchz]

No worries! I know it's working

[sporky_rat]

Dreamwidth did two sales of seed accounts - one when they first opened the doors and one when there was the kerfluffle with payment processors. First was for seed money (more or less) and the second to replenish the emergency fund. Unless something dire happens, I don't think there will be another sale of seed accounts. denise had a good write-up of why it's not in the best business sense and once I'm not on my mobile, I'll see if I can't find it. (Gist was a small amount of money every month (six months, year, etc) was more sustainable than one lump sum and no more. But I'll dig out the actual write-up in a bit.)

[denise]

Huh! That's an interesting idea. I'm not sure we have the resources to do it (it would definitely require a lot of research first, since none of us are Tor experts) but I'll put it on the list of Things To Think About In The Future.

[brickhousewench]

Oh heck yeah, not arguing with that. I was NOT happy when I clicked the "not now" button on the TOS and LiveJournal tried to log me out. =(

[yurikhan]

Okay. Hanlon’s Razor strikes again.

[denise]

I mean, I've been working with (some variety of) this codebase for like 16 years now. I KNOW HOW HARD IT IS TO FEEL CONFIDENT YOU ADEQUATELY UNDERSTAND ALL THE DETAILS OF WHAT WILL HAPPEN IF YOU CHANGE THAT ONE LINE. It's eighteen years' worth of legacy Perl! Sometimes I think it's a wonder it still works at all.

(This is why occasionally, we will do a code push, and five minutes later, we will make another code push consisting only of the emergency commit that has a commit message of just "lol oops" or similar...)

[rollingheart]

Ah! *has to stop and wave at the Shep icon*

[rollingheart]

I've here off and on for years, but the last nail has been put in the coffin or however that goes. Thank you so much for always being so communicative with your users!

[burbilog]

Yes, consider this: if you serve native Tor and block Tor exit nodes (these are anyway could be malicious and could do much more harm besides dumb DDoS) then your main system is not going to be affected anymore by Tor attacks. On other hand, if someone somehow DDoSes your Tor subsystem, only that service is going to be down.

From Tor FAQ:

What about distributed denial of service attacks?

Distributed denial of service (DDoS) attacks typically rely on having a group of thousands of computers all sending floods of traffic to a victim. Since the goal is to overpower the bandwidth of the victim, they typically send UDP packets since those don't require handshakes or coordination.

But because Tor only transports correctly formed TCP streams, not all IP packets, you cannot send UDP packets over Tor. (You can't do specialized forms of this attack like SYN flooding either.) So ordinary DDoS attacks are not possible over Tor. Tor also doesn't allow bandwidth amplification attacks against external sites: you need to send in a byte for every byte that the Tor network will send to your destination. So in general, attackers who control enough bandwidth to launch an effective DDoS attack can do it just fine without Tor.

[inkstone]

*drops off some pancit & egg rolls*

Welcome, new folks!

[parityanimal]

Thanks for this timely reminder! If I ever knew importing from LJ was possible, I'd forgotten. So now I've added mine to the queue. Sorry to add to the slowness...but must say it got my icons and friends almost instantly! Good job!

[bsgsix]

I just moved in after almost 15 years on LJ - I simply couldn't agree to their TOS. And I have to admit that it feels more personal here, and like people care about one another as opposed to making tons of cash off every user. I thank you for that. I'm sad about LJ and everything that happened, but am happy I could move my entries here and simply (as well as safely) begin again. So thanks for caring about us and welcoming us to our new apartment spaces. All of the kind communication is a very welcome change!

[denise]

Thanks for the idea! 

[soc_puppet]

Ugh, likewise. Mine was an early high school graduation present to myself. I will say that it helps me to remember that Dreamwidth is a sit I want to actively support with more money, but I do still feel the loss.