LiveJournal slowdowns/DDoS affecting importer, crossposts

[denise]

Importing

As I'm sure many people have noticed, the waiting period for our content importer is incredibly long right now. Usually, an import will take minutes; right now, the queue is probably at least 24 hours long, and is only getting longer, not shorter.

This is because most of the import jobs waiting in the queue are requesting data from LiveJournal. LiveJournal has been experiencing a Distributed Denial of Service (DDoS) attack on and off since March 24. This means that the site is often unreachable, either because of the network traffic brought about by the DDoS or because of tactics being taken to block the network traffic brought about by the DDoS. (In other words, sometimes being unable to reach LJ is because of the clogged network, and sometimes it's because of a 'false positive' identification as part of the problem and a temporary block.)

These temporary blocks and timeouts when accessing LJ affect the importer (and the crossposter!) as well. The importer and the crossposter both try multiple times before giving up, and each of those tries takes time. Sometimes an import job will get through, because it hits LJ at a time when traffic is lower. More likely, the import job will loop through its scheduled attempts, then fail.

What this means for you:

* Trying to import your journal right now is extremely delayed, and will likely only get worse. Please consider waiting a few days or a week to try to import your journal from elsewhere.

* If you do start an import job, it will take a very long time to reach the front of the queue. You will not receive any results from your import for at least 24 hours or so. It's okay to close the window or tab where you started the import. You'll receive a notice in your Dreamwidth Inbox when the job runs.

* It is very, very likely that your import job will fail. If your job reaches the top of the queue at a time when LJ is unreachable, the job will retry a few times, then error out. Please consider waiting a few days or a week to try again.

* Each individual step of the import runs under its own separate job. This means that some of your data may import while other bits won't. When the problem calms down, you can re-run your import and only bring in the data that it missed. (You can run an import as many times as you want. Each time will only pick up new content; it won't result in duplicates.)

* Edit, because people have asked: There's no way to cancel an import job once you start it. It's all right, though -- you can close the tab/window and log off, and the process will just stay in the queue until it's your turn.


Crossposting

Crossposts are also occasionally affected by LiveJournal being unreachable. If you try to crosspost to LiveJournal and the job fails, you'll receive a notice in your Dreamwidth Inbox (it should say something about the XML-RPC interface being unavailable). Once the problem calms down, you can "retroactively" crosspost the entry by using the Edit Entry link, making sure the crosspost option's box is ticked off, and save the entry. You don't need to make any changes to the entry itself.


Our fingers are crossed for LJ that they can overcome this problem soon! (Hopefully before their operations team has a nervous breakdown.)

Comments

[illariy]

(You can run an import as many times as you want. Each time will only pick up new content; it won't result in duplicates.)

The importer is truly awesome. \o/

(Hopefully before their operations team has a nervous breakdown.)

I kind of want to send the LJ ops team cupcakes but teh logistics make it impracticable.

[archangelbeth]

Thank you for the update! I now want to run the importer to pick up some new stuff, but I will wait till I've forgotten about it a few times, first. O;D

[yeuxdebleu]

Thanks for the update. Compounding the problem might be that the LJ servers are noticing the increased requests from DW and blocking them, as they've been doing with requests that were spambots. I'm no techie so I'm just guessing.

It's good that you're keeping us informed because so many LJers are coming here now. We don't want you to get bad press for something that's beyond your control. Keeping members informed is one of the smartest things you can do.

[azurelunatic]

I'm going to retry next week; I made cookies but the logistics were ow. They're good cookies, though. I could make another batch. This time with enough flour.

[denise]

No, LJ isn't intentionally blocking requests from us! Any blocks are just collateral side damage as part of the DDoS. We follow all their "bot rules" for accessing the site's data feeds, which among other things include a requirement to include identification and contact information for who operates the "bot". We've gotten blocked a few times, but each time was a misunderstanding/miscommunication that was quickly cleared up.

[denise]

See, this is why we need an improved to-do list :)

[azurelunatic]

One thing about the crossposts: if you've posted since then and it did go through, you'll have to do some date-fiddling or use the appropriate checkbox to make it Dated Out of Order on LJ. I had some problems with that.

[oraien]

I think a lot of people are worried about LJ taking the final dive and taking all their information with it.

They could try using something like LJ-Sec, that might be a little less stressful on both LJ and DW while the issues go on? I've had a lot of success with the newest version of LJ-Sec, even in moving content from my private writing comm on LJ over to my DW site, which was super convenient.

Thanks for the updates on the importer and cross-poster. I know I was a bit miffed when my CP failed even though LJ was up and pretty well functioning at full speed recently.

[denise]

The problem is that the DDoS mitigation efforts LJ is taking involve blocking/filtering traffic that may be part of the DDoS, and there's a high rate of false positive involved (I know a bunch of people who can't reach LJ because of it!) It's certainly better than the site being down for everyone, but it can result in things like what you've observed, where you can reach LJ but Dreamwidth's servers can't (or vice versa!)

LJ-Sec isn't any less of a load on LJ's servers than DW's importer is -- it uses the exact same method for obtaining data, in fact -- but it certainly would ease the load on DW's importer!

[yeuxdebleu]

Hey, thanks for the info. It's interesting since, as I said, I'm no techie. :>)

[denise]

No worries! We never mind explaining things :)

But seriously, the only thing to do is to sit tight and wait things out. If the DDoS controllers are renting botnet time to perform the DDoS, they'll run out of money eventually. If they control the botnet directly, eventually they will slip up and do something that will let LJ filter the malicious traffic. It will cost LJ a lot of time and money, but they know what they're doing, and they're probably among the most skilled people at DDoS mitigation on the web! (They get a lot of practice.) In the meantime, the best thing to do is to reduce the amount of traffic you provide to LJ (to help lower the load overall), make sure that everyone knows this is likely an act of political terrorism and not LJ's fault, and be patient.

[ursamajor]

Hooray, somebody's maintaining the fine tradition of cookiesforlj :)

[tifaching]

Good to know importing more than once won't result in duplicate entries. Thanks, that's awesome!

[yeuxdebleu]

make sure that everyone knows this is likely an act of political terrorism and not LJ's fault, and be patient.

Oh, I've definitely been doing that. I'm telling people at LJ, here and in emails. I've also been asking them all to wait a week or so to try to import, because I know many of them plan to do that. I'm been linking them to your update posts as well.

I have a permanent LJ account so it's not going anywhere, nor do I intend to import. This is my "new house." I don't want it cluttered up by my old drivel. LOL Besides, every entry I ever considered important was written in Word and saved.

It's heartbreaking what the LJ team is going through. They must be absolutely exhausted by now. You're probably pretty tired, too, trying to avoid collateral damage.

Keep the faith...and thank you!

[amoureux_politique]

Any idea how/if imports will be affected by the upcoming server maintenance? Thanks!

[denise]

The maintenance will just affect the ability to connect to DW (and DW's ability to connect to the outside world) for about 15 minutes or so, so at most it would make any jobs that run during that time period fail. We'll probably pause the queue for the maintenance window in question.

[amoureux_politique]

Thanks for the info! I was just curious. :-)

[yukinoomoni]

Before all of this happened, I tried to run the importer, but it kept crashing about a fifth through. Is there a limit to how many years or entries it can import? (My LJ has almost 5500.)

[foxfirefey]

No, that shouldn't be a problem--if (after this dies down) you try and it still keeps crashing, go to Support.

[denise]

There can be some problems with large imports timing out, but 5500 entries isn't really enough to qualify as a large import. If you tried anytime between March 24 (when the DDoS was first identified) and now, you probably were being affected by the DDoS. Try again in a week or two, and if you still have problems, report the issue to Support, including the text of the failure message you get in your inbox.

[reportergirl]

This is awesome, thanks so much for addressing this issue! I was wondering why it was taking so long. This is truly an awesome feature! :)

[montuos]

(You can run an import as many times as you want. Each time will only pick up new content; it won't result in duplicates.)

No dupes? That is totally totally awesome! I somehow missed that when I read the FAQ. I will now plan to import again after this mess settles back down, 'cause it took me a while to get in the habit of crossposting from here instead of posting from LJ, and I never did post everything here even then. :)

What happens if you have posts you imported a while back that have since been updated on either DW or the original site? Does the importer somehow remember them and therefore skip over them, or does it reimport differing content?

What about if you did not originally import icons (permanent LJ account to free DW account was not going to be pretty!); will a re-import correct the icons on the posts?

What about custom friend groups after you've friended, de-friended, and switched people around in your filters? Will those update to reflect the current membership, or just add any new filters?

[denise]

1) The importer remembers them and skips them. It does not import the changes from LJ (to avoid overwriting cases where the changes were on DW).

2) Yes.

3) Just add any new filters. Again, this is to avoid overwriting changes -- the system doesn't know which version is the 'correct' version, so it doesn't even try.

[josilverdragon]

I've been trying for about 2 days now, partially because I'm worried (a tad) about losing the entries there. I'm just going to have to wait it out or do it all manually and I'm busy enough as it is.

Thanks for the update. I'll be patient and retry next week.

[highlyeccentric]

Now, here's a question: if I run the importer (in a week or a month or something), will it import comments which have been posted on LJ to entries cross-posted from DW?