Crossposting to LJ throttled; LJ imports slow/failing

[denise]

As LJ has now officially confirmed that they're experiencing a DDoS attack, and a pretty nasty one at that, we've throttled the rate at which we attempt to crosspost entries, and reduced the number of times the process will try before it fails. (It was five tries at an interval of 10 seconds, 30 seconds, 60 seconds, 5 minutes, and 10 minutes; now it's three tries at 1 minute, 5 minutes, and 15 minutes.)

You don't need to do anything differently when posting your entries, but if you receive a failure notice after the third try (which should happen around 20 minutes after you try to crosspost your entry), wait a while and then try to crosspost again, by editing the entry, making sure the crosspost box is checked, and saving the entry.

If the crossposting continues to fail after that, just hang tight and wait a day or so before you try again. LJ's operations and engineering teams are doing everything they can to restore access to the site, including restoring the ability to post from elsewhere using the LJ protocol (which is what the Dreamwidth crossposter uses).

Importing from LJ is also affected by this, obviously, which is driving up the import queue. If you can, please consider waiting a few days to import your journal from LJ -- the process is likely to fail right now anyway. If you get a failure in your inbox when trying to import, just hang tight and try again in a few days.

If the traffic situation at LJ continues to worsen, or if the backup gets too large, we may temporarily disable LJ as an import source, in order to let the queue clear out a bit. If that happens, I'll post here about it again!

We wish LJ and the LJ team the absolute best of luck in dealing with this situation, and we hope that they can recover quickly!


EDIT: And the slowness on our end that some people were noting turned out to be another stuck process and Apache in general sucking up too much memory. alierak is monitoring and fixing.

EDIT #2: (1) The problems on our end involve some bug that's causing the webservers to work too hard that we're trying really hard to chase down the source of. If DW is slow or sluggish for you, that's the cause. We've put some measures in place to deal with it temporarily, and we're working on tracing down the root source of the problem. It's not traffic-related!

(2) If you comment, please try to keep it on the topic of Dreamwidth, not LJ -- we'd like to avoid the comments turning into a LJ-bashing session. Thanks, all.

Comments

[denise]

The importer keeps the same security level on DW as the entries had on LJ, unless your journal's minimum security level is set to a more restrictive level. If you wanted to import everything as "private", you'd set your journal minimum security to private, run the import, wait until the import was done, and then set the minimum security back to whatever you wanted it to be. (I actually did that with another journal of mine; it was very useful!)

[denise]

I've posted a followup with more information! The problem is that a bug in the code is causing memory leaks, which is causing our webserver processes to lock up after a certain amount of time. Under normal load, the length of time it took to lock the server was longer than the length of a time a process usually runs before getting killed and restarted, so the leak wasn't noticeable, but as traffic increases, the processes are growing out of control faster than they can be recycled. That's what causes the periodicity of the problem.

We're working on figuring out the source of the memory leaks, as well as getting some more resources into the webserver pool to help cover for the fact that we've had to lower the number of processes running and lower the amount of time they "live" before getting recycled. We're already seeing some improvement, and I'm hoping there'll be more soon.

[denise]

archangelbeth is correct! If you've already imported your journal and then run another import later, the importer will only bring in new posts and comments that were made since the last import. If you've been using the built-in DW crossposter to crosspost your entries, the importer won't duplicate entries; it'll just put the comments on the LJ-side on the DW post. (If you were crossposting manually by cutting and pasting, or using a client's crossposting ability instead of crossposting through DW, you unfortunately will wind up with duplicate posts, because the importer can only recognize duplicates if they were posted using the DW crossposter.)

[denise]

Your memory is correct! :)

[ovalencia]

I imported my journal a bit after the last huge outage that LJ suffered and I've been using dreamwidth more and more lately, which is awesome because I've been looking forward to moving over for a long time anyway.

The thing is, my friend and I have a personal community that's closed to other members, where we post our ramblings or whatever. I was wondering if you could import communities the way that you can import personal journals. We're both here on DW and were hoping that we could save our content.

[denise]

You and me both!

[denise]

Importing communities is a harder problem than importing personal journals, since communities can be a collective work by a lot of people, some of whom might not want their entries moved. We work around that problem with importing comments by importing them as OpenID identity comments, which means that people can log in using OpenID and delete the comments, but OpenID users can't (currently) make posts to communties, which means that we can't use that solution for community posts as well. The answer to that is to change things around so that OpenID users can post to communities (as I believe LJ has recently done) and then import the community with posts being posted by the OpenID equivalent of the poster, just like comments are imported, but we've had to solve a bunch of other problems first and put in a bunch of backend work before we can do that. (It's a case of technical debt in action.) We also have to figure out some practical solutions -- like, if there are multiple admins, should every admin have to give permission, or should any admin be able to import? Should it be possible to import a community into a personal journal? Should it be possible to import a personal journal into a community? Etc, etc, etc. (All of those questions make it more complicated.)

It is something that we're working on, and I believe the person who the bug is assigned to has gotten about halfway, but it's not as simple as it seems!

[ovalencia]

These are all of the things that ran through our minds while we were trying to find out if we could import. There are only two admins and we're the only members so it would've been perfect and easy for us!

I hope that it gets worked out because it would be pretty awesome, though I def see why it's not readily available. Thanks for the answer!

[gamgeefest]

Thank you, Denise!

It took awhile, but the import did finally work! Guess LJ is still being a little quirky.

[kore]

....OMG, now I also have purple. Thank you!

[musyc]

XD You're welcome!

[musyc]

XD Glad you found it!

[musyc]

:DDD *\o/* Purple siteschemes FTW!

Re: For LJ refugees: comment pages

[kore]

Dang, that's awesome, thank you. Now I can read comments and I have a great purple site skin!

Re: For LJ refugees: comment pages

[denise]

FYI, if you want to set how your comment pages (or your community's comment pages) appear to others, you do that at the Customize Options page. Under "Basic Options", uncheck "Show entry pages in my journal style rather than the site layout".

[triadruid]

Thanks. I frigging HATE memory stuff like that...

[lannamichaels]

Thanks. Very useful information. :D

[azurelunatic]

Turtles all the way down! :D

[shanghaied]

Like there was ever any doubt? ;-)

[lannamichaels]

When importing entries that were crossposted, and with minimum security set to locked, does this switch the security on those entries to access locked if they were public originally?