Crossposting to LJ throttled; LJ imports slow/failing

[denise]

As LJ has now officially confirmed that they're experiencing a DDoS attack, and a pretty nasty one at that, we've throttled the rate at which we attempt to crosspost entries, and reduced the number of times the process will try before it fails. (It was five tries at an interval of 10 seconds, 30 seconds, 60 seconds, 5 minutes, and 10 minutes; now it's three tries at 1 minute, 5 minutes, and 15 minutes.)

You don't need to do anything differently when posting your entries, but if you receive a failure notice after the third try (which should happen around 20 minutes after you try to crosspost your entry), wait a while and then try to crosspost again, by editing the entry, making sure the crosspost box is checked, and saving the entry.

If the crossposting continues to fail after that, just hang tight and wait a day or so before you try again. LJ's operations and engineering teams are doing everything they can to restore access to the site, including restoring the ability to post from elsewhere using the LJ protocol (which is what the Dreamwidth crossposter uses).

Importing from LJ is also affected by this, obviously, which is driving up the import queue. If you can, please consider waiting a few days to import your journal from LJ -- the process is likely to fail right now anyway. If you get a failure in your inbox when trying to import, just hang tight and try again in a few days.

If the traffic situation at LJ continues to worsen, or if the backup gets too large, we may temporarily disable LJ as an import source, in order to let the queue clear out a bit. If that happens, I'll post here about it again!

We wish LJ and the LJ team the absolute best of luck in dealing with this situation, and we hope that they can recover quickly!


EDIT: And the slowness on our end that some people were noting turned out to be another stuck process and Apache in general sucking up too much memory. alierak is monitoring and fixing.

EDIT #2: (1) The problems on our end involve some bug that's causing the webservers to work too hard that we're trying really hard to chase down the source of. If DW is slow or sluggish for you, that's the cause. We've put some measures in place to deal with it temporarily, and we're working on tracing down the root source of the problem. It's not traffic-related!

(2) If you comment, please try to keep it on the topic of Dreamwidth, not LJ -- we'd like to avoid the comments turning into a LJ-bashing session. Thanks, all.

Comments

[naraht]

Thanks for keeping us updated on this!

[keris]

Thanks! Yes, I noticed the slowness (and hoped it wasn't an attack of your own), but ... I couldn't get in to post about it! I don't see any way round that problem...

[pinkadot89]

I assumed the slowness was caused by an influx of angry LJers. :P

I'm glad you're working on it though!

[digitalsidhe]

Thank you to the Dreamwidth team for all your hard work on this... and for keeping us updated! (I'm sure the LJ team would like to keep us updated, but since we can't seem to load any pages over there... heck, they might be posting updates every 15 minutes. I have sympathy for them.)

[hellkitty]

Actually LJ's been awful about updating. We had to discover about the DDOS through other sources. HOURS later they confirmed it. Their lack of communication, honestly, is irritating more LJ users than the DDOS. People are patient when they know what they're being told to be patient for.

[marilla_pm67]

Thank you for telling us LJ is under a DDOS attack .. at least now we know.
And on my end, the site (DW) is still very slow, BUT loading, that's a plus:)


Thanks
sam.

[nightdog_barks]

Thank you for keeping us updated and in the loop! ♥

[cmcmck]

Many thanks.

I've just got in from work and wondered why you were running a bit slowly. I was concerned you were being hit with the same problems as LJ!

[ninetydegrees]

Thank you very much for the update on DW's slowness. And virtual cookies to alierak!

[olchik]

could you help point me to a resource that would help import entries from LJ? I would also like to backup my LJ with comments - do you know who makes that possible?

Thanks for the update!

[denise]

You can import your journal, with comments attached, to your Dreamwidth account (although please do wait a few days until LJ is no longer under so much load), or you can use a program like LJ Backup (if your computer will run it) or a service like LJBook (which turns your entries and comments into a .pdf file for you)!

[le_monde]

Infrequent LJ updates can be found on Twitter, and you don't have to be a registured 'twit' to see it. :)

[surrey_sucks]

Thanks for the update. I am very concerned that LJ might die, so I will be importing my entries as a backup, but I will wait a while until things calm down.

[desds]

After this latest attack I've decided to leave LJ all together and use DW as my permanent journal site. Thanks for keeping us in the know.

[glittertine]

Thanks for the update! :)

[alisanne]

Thanks so much for the update. :)

[phoenixfire12]

And here I thought Frank was taking Mimi out for an extra special date including tasty LJ wiring. Thanks fro keeping us informed. I
ll have to look for their Facebook page and friend them.

[falcon_horus]

Thanks for the update - that explains a lot. :)

[everchangingmuse]

Thanks for the updates, denise! I don't cross-post often, but I'll wait on doing that for anything I might want to in the next few days, just in case. That can always be done with edits, right?

[denise]

Yes! You can edit the post after the fact and save it, making sure that the crosspost box is checked, and the system will try to crosspost it at that point even if you didn't have the crosspost box checked at the time of posting. So, you can just uncheck the box for the next few days at the time of posting, then go back and edit the entries to ticky the tickybox and have them crosspost then.

[kittydesade]

Is there anything us regular users can do to help with tracking down the bug, keep track of what sites are loading slower than others, or anything like that?

(Or, you know, not use DW for a bit, but you know us and how likely that is...)

[denise]

*giggle* No, don't worry about not using DW -- we're on top of things as much as we can be and as soon as we spot the problems, we're fixing it. (Well, I say 'we', when what I really mean is "fu and alierak." I do not know my /dev/arse from my /bin/elbow when it comes to sysadminly stuff.)

This is the kind of thing where the only way to figure it out is to carefully comb through the code and the process trace information to see where the "sticky" bits are. It'd be easier to figure out if we'd just had a code push -- the answer would be "something in the code you just pushed" -- but in this case it's been over a month since our last code push, so something is happening that's only triggering at certain times. We've got some working theories, but the first few didn't pan out, so it's back to the drawing board!

There isn't anything you guys can do to help, though -- although, thanks for asking; y'all are the bestest users in the world. This seems to be one of those annoying bugs that only manifests under load (so it can't really be tested for in advance) and only in certain circumstances (like when the moon is in the seventh house) and those are always a bear to track down.

[musyc]

Thanks for the update, denise. I really appreciate the efforts that the DW code-staff makes to identify/alleviate server stress so quickly as well as the communication and transparency from the admin-staff. I rarely experience downtime here, and I know that it if it's not a shortlived inconvenience, updates will be provided quickly and professionally. ♥

[denise]

Thank you :) We're just lucky this happened when I was awake! (I was actually thinking about going to bed when the slowdowns started, heh.)

[terrierlee]

Yes, and I noticed that DW was kinda slow yesterday too.

But I'm glad LJ and you guys are on top of things.

[brewsternorth]

Thanks for the update on the reason for slowness - I was rather worried there. Looks like whatever your fix was meant to fix fixed as it should!

[swordage]

It might be helpful to make a post about the slowness of DW rather than adding it as edits to this post? I've been seeing a lot of people on Twitter wondering what's up with DW, and the very abbreviated version here still leaves me wondering. Thank you for keeping us updated, though! I understand if you want to hold off on another update until you have a better idea of what's going on. It just seems very opaque at the moment.

[denise]

Yeah, I just didn't want to make yet another post in dw_maintenance when we'd had two posts in very quick succession (especially since the first post mentioned last night's issues already!) Once we get a handle on what's causing the problems, though, we'll definitely make a followup post.

Right now we've got measures in place to keep the problem from getting too bad -- things might slow down a bit from time to time, but they'll come back within a few minutes. (If it seems like you're always hitting the site when it's slow as hell, it's just horrible timing! The CPU activity graphs look like a fairly regular sine wave, heh.) alierak is really digging into things, though, and I'm hopeful that he'll be able to trace down the root cause fairly soon.

[auronlu]

denise, I really really appreciate the fact that DW lets us keep ties with a place where a lot of us have ten years of content/friendships... you don't HAVE to do this, and it's a great perk.

One request which I'm not quite sure where to file.

I am terrible about filtering through all my "inboxes" from email to PMs to Tweets, and I look for as many ways as I can to trim the "fire hose" down to essentials.

Wish list: a way to turn OFF LJ cross-post notifications from landing in our inbox.

I trust DW to do what it can to get that cross-post over there, and for me, personally, it's easy enough to double-check. It's particularly a problem since I'm a nitpicker and tend to edit posts several times to get them "perfect." Each one means another notification. I find it hard to dig through all the cross-post notifications looking for real messages, and I wind up not doing so.

Thanks!

[denise]

We have a bug open for this -- both for a way to filter the inbox only to crosspost (and other system) notifications, for easy finding and deleting, and for a way to only be notified on failure, not on every success as well. I don't know when someone will have a chance to implement it, but we're concentrating on "usability fixes" for the month of August, so I'm hoping someone will pick it up soon!

(Also, I should say: you can filter the inbox to only entries and comments, which will let you see just the stuff that is more likely to need your attention.)

[ironed_orchid]

Thanks for update, especially about the slowness here.

[crivelli]

Perhaps, it's off topic here, but I don't know where can I ask this kind of question. I have difficulties with my communities (those I created), I cannot change the theme. I get the response "your theme is changed", but nothing is changed at all.

[denise]

Try clearing your browser cache -- that's usually the problem in cases like that.

If that doesn't help, open a Support request -- they'll be able to help you troubleshoot!