New imports temporarily stopped; import queue paused

[denise]

The situation with LJ's DDoS affecting our import queue has only gotten worse since my last post, and delays/slowdowns from LJ means that the queue will not clear anytime soon (due to the length of time each job takes to complete or fail with the extremely slow connections we are experiencing). At this point, it's also possible that the traffic from DW's importer is making LJ's situation worse, which we absolutely do not want to do!

We are therefore temporarily pausing the existing import jobs until the traffic clears up some and we are able to further modify the importer so that it is even less of a load on LJ. We are also temporarily stopping the ability to queue new imports, so that once we bring the import queue back up, it will give the queue the chance to clear a bit. (Also, we'd like to minimize the impact on LJ.)

We will re-enable the existing import jobs in the queue once we are able to make some changes to the importer in order to lower the risk that we're making things worse for LJ even more (reducing the number of times a job will try before it fails, further rate-limiting our queries). At that point, we will let the existing jobs in queue run for a while until we can clear out the queue a bit. Once the queue clears, we will re-enable the starting of new jobs.

It will be at least a few days until we are able to re-enable new imports.

If you already have an import job in the queue, you don't need to do anything. It will be automatically re-started when we bring importing back online. (It's not possible for you to cancel an import once it's started -- sorry!)

If you're importing your journal because you want a backup of your LiveJournal, you may want to try the LJ Archive software, which will give you a local copy of your journal. (You can't upload it into your DW account, but at least you will have the backup!)

If you're trying to import from a source other than LJ, I'm really sorry that the LJ DDoS slowdowns are affecting you! We're working right now on the ability to only pause imports from a single source, rather than all of them. Once we have that, we'll re-enable imports from sites other than LJ, and let the LJ imports sit in the queue until LJ is feeling better.

I'm really sorry about all the inconvenience!

EDIT, 2PM EDT 4/10/11: We have just pushed live a change that will allow us to re-enable the queuing of imports. Imports from sources other than LJ will now be processed as normal. Imports from LJ will be held in the queue and not processed until things calm down a bit on their end.

You can therefore submit an import request to import from LJ and forget about it. We will run the LJ imports once the DDoS and resulting issues clear up a bit on their end -- it could be a day or two, it could be next week; we just don't know!

Comments

Standard response to network congestion

[hendrikboom]

The normal way to respond to network congestion is exponential backoff. If you're having trouble reaching a site, try again and again, but wait longer each time (say twice as long, or maybe 50% longer or some other multiplicative factor) before you try again. That way you automatically reduce the load when things are tight. And reduce the delay when you do manage to connect, of course.

Re: Standard response to network congestion

[denise]

Yes, our importer workers already have a built in delay over five tries, the first being 10 seconds and the last being 10 minutes. (After the fifth failure, it aborts the job.) Evidence over the past few days indicate that isn't enough, and most of the jobs are failing -- hence, the complete pause of imports from LJ.

[morri_gun]

just wanted to say thnx for what you are doing. so, thnx )

[amara1783]

This doesn't affect me directly, but thank you for your clear communication and staying-on-top-of-the-issueness. Which is just awesome :) Makes DW a really cool place to hang out.

[impromptu]

Thank you for doing this. I'm really impressed. That much pressure and you are doing your job beautifully. God bless!
Большое спасибо!

[tygati]

Here's a quasi-related question... can DW import from Deadjournal?

[foxfirefey]

If I remember correctly, their code is a wee bit too old for our importer code to work with--however, their news says they're working on a code update, and when that happens it should be possible.

[tygati]

I shall cross my fingers and hope for the best, then. ^_^

[ossobuco]

Just want to add to the chorus and say that you guys are outstanding. Thanks for the communication, and the constant kindness and civility towards your peers! <3

[fyreharper]

Ahaha, I saw your icon and went "Hey, I know someone who would love that icon"... and then I noticed the username ;p

[elialshadowpine]

I'm going to say it again: you guys are pure awesome. I am really very impressed with how you've been supporting LJ through all of this hell. Thank you guys.

[elialshadowpine]

They mentioned in the last post that cross-posts could be affected, yes. :(