Crossposting to LJ throttled; LJ imports slow/failing

[denise]

As LJ has now officially confirmed that they're experiencing a DDoS attack, and a pretty nasty one at that, we've throttled the rate at which we attempt to crosspost entries, and reduced the number of times the process will try before it fails. (It was five tries at an interval of 10 seconds, 30 seconds, 60 seconds, 5 minutes, and 10 minutes; now it's three tries at 1 minute, 5 minutes, and 15 minutes.)

You don't need to do anything differently when posting your entries, but if you receive a failure notice after the third try (which should happen around 20 minutes after you try to crosspost your entry), wait a while and then try to crosspost again, by editing the entry, making sure the crosspost box is checked, and saving the entry.

If the crossposting continues to fail after that, just hang tight and wait a day or so before you try again. LJ's operations and engineering teams are doing everything they can to restore access to the site, including restoring the ability to post from elsewhere using the LJ protocol (which is what the Dreamwidth crossposter uses).

Importing from LJ is also affected by this, obviously, which is driving up the import queue. If you can, please consider waiting a few days to import your journal from LJ -- the process is likely to fail right now anyway. If you get a failure in your inbox when trying to import, just hang tight and try again in a few days.

If the traffic situation at LJ continues to worsen, or if the backup gets too large, we may temporarily disable LJ as an import source, in order to let the queue clear out a bit. If that happens, I'll post here about it again!

We wish LJ and the LJ team the absolute best of luck in dealing with this situation, and we hope that they can recover quickly!


EDIT: And the slowness on our end that some people were noting turned out to be another stuck process and Apache in general sucking up too much memory. alierak is monitoring and fixing.

EDIT #2: (1) The problems on our end involve some bug that's causing the webservers to work too hard that we're trying really hard to chase down the source of. If DW is slow or sluggish for you, that's the cause. We've put some measures in place to deal with it temporarily, and we're working on tracing down the root source of the problem. It's not traffic-related!

(2) If you comment, please try to keep it on the topic of Dreamwidth, not LJ -- we'd like to avoid the comments turning into a LJ-bashing session. Thanks, all.

Comments

[arethinn]

The cool thing about the crossposter is that it maintains links between the original on DW and the remote posts. If you edit or delete the DW copy, the same happens to the LJ (etc) copy without you having to do anything manually.

[olchik]

Wow, this is an awesome feature! I guess I will spend more time here from now on. Now, do all crossposts post with the same privacy settings as your dw posts or do you have to edit your privacy settings in LJ after posting?

[alyse]

They haven't exactly been good about using Twitter or Facebook either, with updates there being about as frequent as changing the status page.

[denise]

It posts with the same security you used for the post on DW! The one exception is if you've set a journal-wide minimum security level of "access only" (or "private") for your DW journal, but haven't set a journal-wide minimum security level for your LJ journal. If you then post an entry on DW as "public" (that is, you don't change the security setting from the default when you're posting), it will post locked on DW (applying your journal-wide minimum security), but post public on LJ. The solution there is to set the minimum security level on both journals to be identical.

(Of course, the reverse is true: if you have a journal-wide minimum security setting of "friends" on LJ, but don't have a journal-wide minimum security set on DW, and then you post something with a security setting of "public" to DW, the version posted to LJ will obey your minimum security level and be posted as friends-only.)

[magid]

It's the same public/private/friendslocked, but I haven't checked whether it works for filters with the same name.

[denise]

Let's try to keep this from turning into a "let's bash LJ" comment thread, okay? Many of the people who work there are friends of ours, and I know for a fact that the people who are responsible for updating the various status/communications channels are just as frustrated by being unable to get accurate information from others within the company to pass on.

[olchik]

As in groups?

[kate_nepveu]

Yes, except if you use custom groups, you have to make sure the custom groups have the same names both places (otherwise, IIRC, the custom-DW post is made private on LJ).

[magid]

Yes, but it looks like Kate_Nepveu answered that :-)

[olchik]

I guess now we know :)

[alyse]

I don't think that a single mild comment in response to someone other than you counts as bashing, personally, but please feel free to single me out.

[glittertine]

Thanks for the update! :)

[denise]

Let's try to keep this from turning into a "let's bash LJ" comment thread, okay? Many of the people who work there are friends of ours, and I know for a fact that the people who are responsible for updating the various status/communications channels are just as frustrated by being unable to get accurate information from others within the company to pass on.

[alisanne]

Thanks so much for the update. :)

[denise]

I'm sorry that you felt I was singling you out! I've been making similar comments elsewhere when I see people making similar comments -- I'm just a little slow on some of them, since I'm also juggling trying to help our sysadmin team troubleshoot the problem we're having with our Apache processes using way too much memory.

It just makes me very sad when I see comments about how much LJ sucks in response to posts here, since a whole lot of people try to turn "LJ vs DW" into an either/or, and sometimes it feels like people can't enjoy DW without expressing frustrations about LJ. I am far from a fan of some of the decisions LJ's management has made over the past few years and I obviously think DW is a superior product for many use cases, but I also think that LJ is a superior product for other use cases (which, obviously, many of our users aren't fans of, or they'd be over there instead of over here!) and I think it's completely valid to maintain accounts on both sites and enjoy both of them. A lot of people have this perception of DW as being anti-LJ, and not only is that not good for DW, it's personally very upsetting for me, since I started working on LJ ten loooooong years ago and DW literally could not exist without it.

One of the things that bothers me about LJ is the tone of discourse that the comments to their official announcements tend to take on over time due to lack of moderation, and we agreed when we started DW that we would do our best to keep DW's offical communities' comments as polite and respectful as we could. This involves stepping in whenever I see something that has the potential to go downhill, as soon as I can. Sometimes I don't get to something fast enough, and sometimes I'm sure I jump the gun a little, but it's something we feel really passionate about and try to watch out for as much as possible. I don't want to keep people from raising concerns they have with Dreamwidth, but this isn't the place for concerns with LiveJournal, and I don't want the comments to become a string of people complaining about any other site, not just LJ!

[phoenixfire12]

And here I thought Frank was taking Mimi out for an extra special date including tasty LJ wiring. Thanks fro keeping us informed. I
ll have to look for their Facebook page and friend them.

[archangelbeth]

I think it's a boilerplate (see the comment thread above!); it's so easy for cynical LJ-users/ex-users to snowball from one minorly annoyed comment (or statement of objective fact, even...) to a full-on LJ-bash, that it's probably better to nip it in the bud.

Otherwise, I might start making snarky comments like **COMMENT INTERCEPTED BY ANTI-BASHING PROTOTYPE SOFTWARE**


...dang, they're good, eh? O_O

[archangelbeth]

Well, you have to double-check the tickybox to cross-post the edit, but that's all. (I love being able to post from DW (and the DW posting page is only going to get more awesome in the future) and have everything go over to LJ. Even if most of my reading and readers are still on LJ -- DW is worth it to me.)

[denise]

Everything I've seen has indicated that was an overreaction/misinformation by one particular news outlet -- it's really easy for misinformation to spread, especially when dealing with language barriers and iffy translations. I would be really suspicious of any statement like that, since rumor spreads really quickly in a case like this!

At any rate, I don't want this comment thread to turn into a place for people to vent frustrations with LJ -- we like to keep the comments in official DW-related journals on DW-specific stuff.

[hendrikboom]

I think it's perfectly reasonable for LJ to have trouble finding time to update the status page when it's suffering a DDOS attack -- priority has to go to fighting the DDOS. I don't expect a blow-by-blow news chronicle. But as a computing professional, I'd find it very interesting to read if it were to be written up after the fact.

That said, I'm here, and I appreciate your rapid responses here. I certainly appreciate you telling me about the LiveJournal DDOS -- I'm one of the people who maintain accounts in both places, and I've been wondering if my own net connection were somehow being weird.

-- hendrik

[falcon_horus]

Thanks for the update - that explains a lot. :)

[arethinn]

They carry the privacy settings of the original. Everyone/public is Everyone, obvously. If you post to Access List that becomes Friends only on LJ. If you have custom filters, the remote post will inherit them if there is an existing filter of the same name on LJ (like I have a filter called "closer" in both places); if not, I think it defaults to Private (that is, it makes it more secure, not less).

Similarly with icons: if you maintain identical keywords on both sites, the LJ post will have the icon you selected (if no icon exists with that keyword then you get the default icon).

Is verra verra fancy. We loves it, precious.

[everchangingmuse]

Thanks for the updates, denise! I don't cross-post often, but I'll wait on doing that for anything I might want to in the next few days, just in case. That can always be done with edits, right?

[arethinn]

Funny, I don't have to do that. Mine is auto-checked (since I have "automatically crosspost" set) and I don't have to un-check it and re-check it if I actually did make a change in the post. (as opposed to trying to re-crosspost a failed crosspost with no actual edits)

[denise]

Yeah, especially since the industry standard is "do not talk publicly about a DDoS and your mitigation efforts until the mitigation efforts are fully in place", to avoid publicity for the people doing the DDoS (which is, after all, what they want).

Some of the translated news articles I've seen on this have said they're receiving between 6Gbps and 8Gbps steadily -- that's just ridiculous. I'm hoping to see a postmortem as well, since that's pretty much "destroy things at the backbone" level of incoming.