LiveJournal slowdowns/DDoS affecting importer, crossposts

[denise]

Importing

As I'm sure many people have noticed, the waiting period for our content importer is incredibly long right now. Usually, an import will take minutes; right now, the queue is probably at least 24 hours long, and is only getting longer, not shorter.

This is because most of the import jobs waiting in the queue are requesting data from LiveJournal. LiveJournal has been experiencing a Distributed Denial of Service (DDoS) attack on and off since March 24. This means that the site is often unreachable, either because of the network traffic brought about by the DDoS or because of tactics being taken to block the network traffic brought about by the DDoS. (In other words, sometimes being unable to reach LJ is because of the clogged network, and sometimes it's because of a 'false positive' identification as part of the problem and a temporary block.)

These temporary blocks and timeouts when accessing LJ affect the importer (and the crossposter!) as well. The importer and the crossposter both try multiple times before giving up, and each of those tries takes time. Sometimes an import job will get through, because it hits LJ at a time when traffic is lower. More likely, the import job will loop through its scheduled attempts, then fail.

What this means for you:

* Trying to import your journal right now is extremely delayed, and will likely only get worse. Please consider waiting a few days or a week to try to import your journal from elsewhere.

* If you do start an import job, it will take a very long time to reach the front of the queue. You will not receive any results from your import for at least 24 hours or so. It's okay to close the window or tab where you started the import. You'll receive a notice in your Dreamwidth Inbox when the job runs.

* It is very, very likely that your import job will fail. If your job reaches the top of the queue at a time when LJ is unreachable, the job will retry a few times, then error out. Please consider waiting a few days or a week to try again.

* Each individual step of the import runs under its own separate job. This means that some of your data may import while other bits won't. When the problem calms down, you can re-run your import and only bring in the data that it missed. (You can run an import as many times as you want. Each time will only pick up new content; it won't result in duplicates.)

* Edit, because people have asked: There's no way to cancel an import job once you start it. It's all right, though -- you can close the tab/window and log off, and the process will just stay in the queue until it's your turn.


Crossposting

Crossposts are also occasionally affected by LiveJournal being unreachable. If you try to crosspost to LiveJournal and the job fails, you'll receive a notice in your Dreamwidth Inbox (it should say something about the XML-RPC interface being unavailable). Once the problem calms down, you can "retroactively" crosspost the entry by using the Edit Entry link, making sure the crosspost option's box is ticked off, and save the entry. You don't need to make any changes to the entry itself.


Our fingers are crossed for LJ that they can overcome this problem soon! (Hopefully before their operations team has a nervous breakdown.)

Comments

[shutterbug]

Also, I don't know the ins and outs of DDoS attacks, and I know LJ is much more likely to be targeted than DW, but does DW have some kind of protection in place from things like this?

[denise]

Any large web service has a plan for what to do if under DDoS attack! (We've weathered a few of them already.) We don't have the resources for huge major DDoS mitigation, so it's very likely we would be unreachable for part if not all of the duration of the attack, but there are ways to work around things even when you're smaller and have a smaller budget, and our sysadmins are very experienced.

[xoxomarina]

ily

[primwood]

I'm waiting, but I'm very nervous! I hope poor LJ doesn't get totally wiped out before I get a chance to copy it over here.

Thanks for the update-y again. :)

[denise]

I seriously, seriously, seriously doubt LJ is going anywhere. Backing up your data is always a good plan, but doing it because you think LJ is going to shut down tomorrow isn't necessary!

[thena]

Thank you for being so forthcoming with information!

[chratz]

thank you for keeping us all informed. I have to say it is a great thing to do, and it is something I really think LJ could do. so thank you!

[dhamphir]

I have a question. I just checked the queue and show it down to (4) entries and zero everything else. Yet I've not recieved any notifications of a successful import or of a failed import. Is the queue showing practically empty because of something you've done?

[fu]

Yup, it's a side effect of how we're processing the jobs from LJ. Since they're being processed separately and not currently considered active, they don't show up in the queue. That also makes it easier to figure out how long things will take for active imports.

Once we have re-enabled imports from LJ, they'll be counted in the regular queue, and once they're done there notifications of success/failure will be sent.

[dhamphir]

Okay. Thanks for such quick answer! :D