Crossposting to LJ throttled; LJ imports slow/failing

[denise]

As LJ has now officially confirmed that they're experiencing a DDoS attack, and a pretty nasty one at that, we've throttled the rate at which we attempt to crosspost entries, and reduced the number of times the process will try before it fails. (It was five tries at an interval of 10 seconds, 30 seconds, 60 seconds, 5 minutes, and 10 minutes; now it's three tries at 1 minute, 5 minutes, and 15 minutes.)

You don't need to do anything differently when posting your entries, but if you receive a failure notice after the third try (which should happen around 20 minutes after you try to crosspost your entry), wait a while and then try to crosspost again, by editing the entry, making sure the crosspost box is checked, and saving the entry.

If the crossposting continues to fail after that, just hang tight and wait a day or so before you try again. LJ's operations and engineering teams are doing everything they can to restore access to the site, including restoring the ability to post from elsewhere using the LJ protocol (which is what the Dreamwidth crossposter uses).

Importing from LJ is also affected by this, obviously, which is driving up the import queue. If you can, please consider waiting a few days to import your journal from LJ -- the process is likely to fail right now anyway. If you get a failure in your inbox when trying to import, just hang tight and try again in a few days.

If the traffic situation at LJ continues to worsen, or if the backup gets too large, we may temporarily disable LJ as an import source, in order to let the queue clear out a bit. If that happens, I'll post here about it again!

We wish LJ and the LJ team the absolute best of luck in dealing with this situation, and we hope that they can recover quickly!


EDIT: And the slowness on our end that some people were noting turned out to be another stuck process and Apache in general sucking up too much memory. alierak is monitoring and fixing.

EDIT #2: (1) The problems on our end involve some bug that's causing the webservers to work too hard that we're trying really hard to chase down the source of. If DW is slow or sluggish for you, that's the cause. We've put some measures in place to deal with it temporarily, and we're working on tracing down the root source of the problem. It's not traffic-related!

(2) If you comment, please try to keep it on the topic of Dreamwidth, not LJ -- we'd like to avoid the comments turning into a LJ-bashing session. Thanks, all.

Comments

[denise]

Let's try to keep this from turning into a "let's bash LJ" comment thread, okay? Many of the people who work there are friends of ours, and I know for a fact that the people who are responsible for updating the various status/communications channels are just as frustrated by being unable to get accurate information from others within the company to pass on.

[alyse]

I don't think that a single mild comment in response to someone other than you counts as bashing, personally, but please feel free to single me out.

[denise]

I'm sorry that you felt I was singling you out! I've been making similar comments elsewhere when I see people making similar comments -- I'm just a little slow on some of them, since I'm also juggling trying to help our sysadmin team troubleshoot the problem we're having with our Apache processes using way too much memory.

It just makes me very sad when I see comments about how much LJ sucks in response to posts here, since a whole lot of people try to turn "LJ vs DW" into an either/or, and sometimes it feels like people can't enjoy DW without expressing frustrations about LJ. I am far from a fan of some of the decisions LJ's management has made over the past few years and I obviously think DW is a superior product for many use cases, but I also think that LJ is a superior product for other use cases (which, obviously, many of our users aren't fans of, or they'd be over there instead of over here!) and I think it's completely valid to maintain accounts on both sites and enjoy both of them. A lot of people have this perception of DW as being anti-LJ, and not only is that not good for DW, it's personally very upsetting for me, since I started working on LJ ten loooooong years ago and DW literally could not exist without it.

One of the things that bothers me about LJ is the tone of discourse that the comments to their official announcements tend to take on over time due to lack of moderation, and we agreed when we started DW that we would do our best to keep DW's offical communities' comments as polite and respectful as we could. This involves stepping in whenever I see something that has the potential to go downhill, as soon as I can. Sometimes I don't get to something fast enough, and sometimes I'm sure I jump the gun a little, but it's something we feel really passionate about and try to watch out for as much as possible. I don't want to keep people from raising concerns they have with Dreamwidth, but this isn't the place for concerns with LiveJournal, and I don't want the comments to become a string of people complaining about any other site, not just LJ!

[hendrikboom]

I think it's perfectly reasonable for LJ to have trouble finding time to update the status page when it's suffering a DDOS attack -- priority has to go to fighting the DDOS. I don't expect a blow-by-blow news chronicle. But as a computing professional, I'd find it very interesting to read if it were to be written up after the fact.

That said, I'm here, and I appreciate your rapid responses here. I certainly appreciate you telling me about the LiveJournal DDOS -- I'm one of the people who maintain accounts in both places, and I've been wondering if my own net connection were somehow being weird.

-- hendrik

[denise]

Yeah, especially since the industry standard is "do not talk publicly about a DDoS and your mitigation efforts until the mitigation efforts are fully in place", to avoid publicity for the people doing the DDoS (which is, after all, what they want).

Some of the translated news articles I've seen on this have said they're receiving between 6Gbps and 8Gbps steadily -- that's just ridiculous. I'm hoping to see a postmortem as well, since that's pretty much "destroy things at the backbone" level of incoming.

[sara]

I saw that and it makes me really curious about the motivation behind it. What on earth is on LJ that someone wants to knock out that badly?

[lederhosen]

As per comments from the last LJ DDoS, it seems to be an attempt to shut down Russian dissidents who use LJ to expose political corruption.

I've had my share of complaints about the way LJ has been run, but this is definitely not one of them. On this issue, they're paying a hefty price for standing up for free speech.

[denise]

To be scrupulously accurate, information in this case indicates that the attack is against LJ as a whole, not against individual journals the way the last round of attacks were. It may still be politically motivated; it may not. All that anybody knows right now is that someone with a lot of money and a lot of resources is pissed at LJ about something!

[wytchcroft]

well, all your efforts here are much appreciated :)

[lady_ganesh]

And if it isn't specifically about dissidents, it could still be aimed at something specific to Russian LJ users, simply because it's so big in that country.

[maria_gorynceva]

We are going to elect President in 2012. It's safer to keep people silent. LJ in Russia became a powerful resourse to express opinions and to exchange ideas.

But the plot theory can't be proved in this case.

[lady_ganesh]

It's worrying.

[archangelbeth]

I think it's a boilerplate (see the comment thread above!); it's so easy for cynical LJ-users/ex-users to snowball from one minorly annoyed comment (or statement of objective fact, even...) to a full-on LJ-bash, that it's probably better to nip it in the bud.

Otherwise, I might start making snarky comments like **COMMENT INTERCEPTED BY ANTI-BASHING PROTOTYPE SOFTWARE**


...dang, they're good, eh? O_O