Crossposting to LJ throttled; LJ imports slow/failing

[denise]

As LJ has now officially confirmed that they're experiencing a DDoS attack, and a pretty nasty one at that, we've throttled the rate at which we attempt to crosspost entries, and reduced the number of times the process will try before it fails. (It was five tries at an interval of 10 seconds, 30 seconds, 60 seconds, 5 minutes, and 10 minutes; now it's three tries at 1 minute, 5 minutes, and 15 minutes.)

You don't need to do anything differently when posting your entries, but if you receive a failure notice after the third try (which should happen around 20 minutes after you try to crosspost your entry), wait a while and then try to crosspost again, by editing the entry, making sure the crosspost box is checked, and saving the entry.

If the crossposting continues to fail after that, just hang tight and wait a day or so before you try again. LJ's operations and engineering teams are doing everything they can to restore access to the site, including restoring the ability to post from elsewhere using the LJ protocol (which is what the Dreamwidth crossposter uses).

Importing from LJ is also affected by this, obviously, which is driving up the import queue. If you can, please consider waiting a few days to import your journal from LJ -- the process is likely to fail right now anyway. If you get a failure in your inbox when trying to import, just hang tight and try again in a few days.

If the traffic situation at LJ continues to worsen, or if the backup gets too large, we may temporarily disable LJ as an import source, in order to let the queue clear out a bit. If that happens, I'll post here about it again!

We wish LJ and the LJ team the absolute best of luck in dealing with this situation, and we hope that they can recover quickly!


EDIT: And the slowness on our end that some people were noting turned out to be another stuck process and Apache in general sucking up too much memory. alierak is monitoring and fixing.

EDIT #2: (1) The problems on our end involve some bug that's causing the webservers to work too hard that we're trying really hard to chase down the source of. If DW is slow or sluggish for you, that's the cause. We've put some measures in place to deal with it temporarily, and we're working on tracing down the root source of the problem. It's not traffic-related!

(2) If you comment, please try to keep it on the topic of Dreamwidth, not LJ -- we'd like to avoid the comments turning into a LJ-bashing session. Thanks, all.

Comments

[denise]

Yeah, especially since the industry standard is "do not talk publicly about a DDoS and your mitigation efforts until the mitigation efforts are fully in place", to avoid publicity for the people doing the DDoS (which is, after all, what they want).

Some of the translated news articles I've seen on this have said they're receiving between 6Gbps and 8Gbps steadily -- that's just ridiculous. I'm hoping to see a postmortem as well, since that's pretty much "destroy things at the backbone" level of incoming.

[sara]

I saw that and it makes me really curious about the motivation behind it. What on earth is on LJ that someone wants to knock out that badly?

[lederhosen]

As per comments from the last LJ DDoS, it seems to be an attempt to shut down Russian dissidents who use LJ to expose political corruption.

I've had my share of complaints about the way LJ has been run, but this is definitely not one of them. On this issue, they're paying a hefty price for standing up for free speech.

[denise]

To be scrupulously accurate, information in this case indicates that the attack is against LJ as a whole, not against individual journals the way the last round of attacks were. It may still be politically motivated; it may not. All that anybody knows right now is that someone with a lot of money and a lot of resources is pissed at LJ about something!

[wytchcroft]

well, all your efforts here are much appreciated :)

[lady_ganesh]

And if it isn't specifically about dissidents, it could still be aimed at something specific to Russian LJ users, simply because it's so big in that country.

[maria_gorynceva]

We are going to elect President in 2012. It's safer to keep people silent. LJ in Russia became a powerful resourse to express opinions and to exchange ideas.

But the plot theory can't be proved in this case.

[lady_ganesh]

It's worrying.