LiveJournal slowdowns/DDoS affecting importer, crossposts

[denise]

Importing

As I'm sure many people have noticed, the waiting period for our content importer is incredibly long right now. Usually, an import will take minutes; right now, the queue is probably at least 24 hours long, and is only getting longer, not shorter.

This is because most of the import jobs waiting in the queue are requesting data from LiveJournal. LiveJournal has been experiencing a Distributed Denial of Service (DDoS) attack on and off since March 24. This means that the site is often unreachable, either because of the network traffic brought about by the DDoS or because of tactics being taken to block the network traffic brought about by the DDoS. (In other words, sometimes being unable to reach LJ is because of the clogged network, and sometimes it's because of a 'false positive' identification as part of the problem and a temporary block.)

These temporary blocks and timeouts when accessing LJ affect the importer (and the crossposter!) as well. The importer and the crossposter both try multiple times before giving up, and each of those tries takes time. Sometimes an import job will get through, because it hits LJ at a time when traffic is lower. More likely, the import job will loop through its scheduled attempts, then fail.

What this means for you:

* Trying to import your journal right now is extremely delayed, and will likely only get worse. Please consider waiting a few days or a week to try to import your journal from elsewhere.

* If you do start an import job, it will take a very long time to reach the front of the queue. You will not receive any results from your import for at least 24 hours or so. It's okay to close the window or tab where you started the import. You'll receive a notice in your Dreamwidth Inbox when the job runs.

* It is very, very likely that your import job will fail. If your job reaches the top of the queue at a time when LJ is unreachable, the job will retry a few times, then error out. Please consider waiting a few days or a week to try again.

* Each individual step of the import runs under its own separate job. This means that some of your data may import while other bits won't. When the problem calms down, you can re-run your import and only bring in the data that it missed. (You can run an import as many times as you want. Each time will only pick up new content; it won't result in duplicates.)

* Edit, because people have asked: There's no way to cancel an import job once you start it. It's all right, though -- you can close the tab/window and log off, and the process will just stay in the queue until it's your turn.


Crossposting

Crossposts are also occasionally affected by LiveJournal being unreachable. If you try to crosspost to LiveJournal and the job fails, you'll receive a notice in your Dreamwidth Inbox (it should say something about the XML-RPC interface being unavailable). Once the problem calms down, you can "retroactively" crosspost the entry by using the Edit Entry link, making sure the crosspost option's box is ticked off, and save the entry. You don't need to make any changes to the entry itself.


Our fingers are crossed for LJ that they can overcome this problem soon! (Hopefully before their operations team has a nervous breakdown.)

Comments

[illariy]

(You can run an import as many times as you want. Each time will only pick up new content; it won't result in duplicates.)

The importer is truly awesome. \o/

(Hopefully before their operations team has a nervous breakdown.)

I kind of want to send the LJ ops team cupcakes but teh logistics make it impracticable.

[azurelunatic]

I'm going to retry next week; I made cookies but the logistics were ow. They're good cookies, though. I could make another batch. This time with enough flour.

[archangelbeth]

Thank you for the update! I now want to run the importer to pick up some new stuff, but I will wait till I've forgotten about it a few times, first. O;D

[denise]

See, this is why we need an improved to-do list :)

[yeuxdebleu]

Thanks for the update. Compounding the problem might be that the LJ servers are noticing the increased requests from DW and blocking them, as they've been doing with requests that were spambots. I'm no techie so I'm just guessing.

It's good that you're keeping us informed because so many LJers are coming here now. We don't want you to get bad press for something that's beyond your control. Keeping members informed is one of the smartest things you can do.

[denise]

No, LJ isn't intentionally blocking requests from us! Any blocks are just collateral side damage as part of the DDoS. We follow all their "bot rules" for accessing the site's data feeds, which among other things include a requirement to include identification and contact information for who operates the "bot". We've gotten blocked a few times, but each time was a misunderstanding/miscommunication that was quickly cleared up.

[azurelunatic]

One thing about the crossposts: if you've posted since then and it did go through, you'll have to do some date-fiddling or use the appropriate checkbox to make it Dated Out of Order on LJ. I had some problems with that.

[oraien]

I think a lot of people are worried about LJ taking the final dive and taking all their information with it.

They could try using something like LJ-Sec, that might be a little less stressful on both LJ and DW while the issues go on? I've had a lot of success with the newest version of LJ-Sec, even in moving content from my private writing comm on LJ over to my DW site, which was super convenient.

Thanks for the updates on the importer and cross-poster. I know I was a bit miffed when my CP failed even though LJ was up and pretty well functioning at full speed recently.

[denise]

The problem is that the DDoS mitigation efforts LJ is taking involve blocking/filtering traffic that may be part of the DDoS, and there's a high rate of false positive involved (I know a bunch of people who can't reach LJ because of it!) It's certainly better than the site being down for everyone, but it can result in things like what you've observed, where you can reach LJ but Dreamwidth's servers can't (or vice versa!)

LJ-Sec isn't any less of a load on LJ's servers than DW's importer is -- it uses the exact same method for obtaining data, in fact -- but it certainly would ease the load on DW's importer!

[tifaching]

Good to know importing more than once won't result in duplicate entries. Thanks, that's awesome!

[amoureux_politique]

Any idea how/if imports will be affected by the upcoming server maintenance? Thanks!

[denise]

The maintenance will just affect the ability to connect to DW (and DW's ability to connect to the outside world) for about 15 minutes or so, so at most it would make any jobs that run during that time period fail. We'll probably pause the queue for the maintenance window in question.

[yukinoomoni]

Before all of this happened, I tried to run the importer, but it kept crashing about a fifth through. Is there a limit to how many years or entries it can import? (My LJ has almost 5500.)

[foxfirefey]

No, that shouldn't be a problem--if (after this dies down) you try and it still keeps crashing, go to Support.

[reportergirl]

This is awesome, thanks so much for addressing this issue! I was wondering why it was taking so long. This is truly an awesome feature! :)

[montuos]

(You can run an import as many times as you want. Each time will only pick up new content; it won't result in duplicates.)

No dupes? That is totally totally awesome! I somehow missed that when I read the FAQ. I will now plan to import again after this mess settles back down, 'cause it took me a while to get in the habit of crossposting from here instead of posting from LJ, and I never did post everything here even then. :)

What happens if you have posts you imported a while back that have since been updated on either DW or the original site? Does the importer somehow remember them and therefore skip over them, or does it reimport differing content?

What about if you did not originally import icons (permanent LJ account to free DW account was not going to be pretty!); will a re-import correct the icons on the posts?

What about custom friend groups after you've friended, de-friended, and switched people around in your filters? Will those update to reflect the current membership, or just add any new filters?

[denise]

1) The importer remembers them and skips them. It does not import the changes from LJ (to avoid overwriting cases where the changes were on DW).

2) Yes.

3) Just add any new filters. Again, this is to avoid overwriting changes -- the system doesn't know which version is the 'correct' version, so it doesn't even try.

[josilverdragon]

I've been trying for about 2 days now, partially because I'm worried (a tad) about losing the entries there. I'm just going to have to wait it out or do it all manually and I'm busy enough as it is.

Thanks for the update. I'll be patient and retry next week.

[highlyeccentric]

Now, here's a question: if I run the importer (in a week or a month or something), will it import comments which have been posted on LJ to entries cross-posted from DW?

[denise]

It will, yes. the importer and crossposter talk to each other :)

[a_muse_d]

simply, thank you for the update

[elialshadowpine]

Thank you for keeping us updated. :)

I also want to say, you guys have been hella classy dealing with all this. I've seen a lot of blame going toward LJ for the DDoS, although mostly from people who are not techies, and it's really nice to see you guys supporting them even though they are technically your competitors. My husband works as a sysadmin, and I know how stressy DDoSes have been for him at his company. The LJ team has to be going nuts by now. So thank you again. You guys are awesome.

[discodiva76]

Well I must have been incredibly lucky....I started my import a couple of day ago and decided to stay in the queue and see what happened (am going away in 5 days and wanted peace of mind before I went, just in case I came back and found LJ gone for good!)....

My entries successfully imported 3 hours ago.....just waiting in the queue for the comments now...

Thank you DW and LJ....I am very relieved to have backed up my entries at least.....I can head away knowing that 6 years of mindless rambling is safe for a while..

Deeds xx

[vlredreign]

This explains a lot. Thanks for the info.

Question: has this been happening when trying to import from InsaneJournal, or does it not matter where the import comes from because of all the LJ imports being backed up? I want to import my posts from there as well, but I'm gonna wait awhile, I think.

[denise]

It doesn't matter where -- imports go into the queue in order no matter what the source, so your IJ import would be behind the 1127 (at this minute) entries imports from LJ. :)

[samjohnsson]

Not an issue affecting me overmuch, but thank you for keeping us in the loop as to what's going on!

[etfb]

The use of the inbox for error messages is problematical, to put it politely. When you crosspost, you get a message in your inbox. When you cross post every time you post, your inbox fills up. I write a blog post every day, and have done for several years now, so my inbox is full of these redundant messages and as a result is completely and utterly useless for anything else. I'm effectivaly spamming myself, because DW doesn't give an option to switch off those successful messages. Until that's fixed, I might as well assume that DreamWidth doesn't provide any kind of inbox service. That's a pain.

[denise]

You can filter to entries, comments, and private messages to view only those items. We have a bug open to allow users to choose whether or not to be notified on successful crosspost (notifications on failure will always be mandatory), but the notification system is a maze of twisty passages all alike and we need to pay some technical debt before we can change it.

[dimrub]

Thanks for the heads up! I like the way you guys keep the folks up to date. A partially unrelated question, if I may. This latest wave of troubles over in the LJ land caused something of an exodus, and dw seems to be the primary destination for those abandoning ship. What is your policy of giving invite codes to the newcomers? I'd like to invite many of my LJ friends over, but not all of them are comfortable using their credit card to pay for a paid account (a paypal support would have been most welcome).

[denise]

We give out invite codes to our userbase when we feel we have the resources to support more accounts! It usually tends to be once every month or two. We use different criteria to distribute the codes, too -- sometimes we give codes to paid account holders, sometimes to active users, sometimes to people who have used all their invites, etc.

Believe me, we'd love to have PayPal support too. PayPal dropped us last year after an organized troll group created a fictitious concerned-parent organization and reported content on DW to PayPal. PayPal decided that the content -- though perfectly legal and not against our Terms of Service -- freaked them out, so they demanded we remove it as a condition of remaining a customer of theirs. We refused, and PayPal closed our account with them. Payments were totally down for about three months, as a matter of fact, until we could implement our new payment system -- we couldn't take in money except via check/money order during that time.

PayPal is never going to reinstate our account unless we comply with their demands, which we're not going to do, so your friends can check out dw_codesharing, where people share invites they have available. There's also the option of having someone who does feel comfortable using their credit card buy a paid account creation code for them and they reimburse that person through PayPal (or wherever)! We also accept check and money order in US funds, and if cash gets to us we'll apply it to an order (we can't guarantee that it won't get lost in the mail, and there's no recourse if it does, so this is best used for small orders).

[goodbyemyfancy]

Thank you so much for the information! I'm trying to import from LJ and it's been 2 days now, so I think I'll try to stop it and redo it in a couple of weeks. Wanted to use my account here as a back up to the insanity over LJ but when everyone does it at the same time, it's certainly changes things for DW!

[denise]

You can't cancel an import once it stops, but it will either finish or abort eventually!

And the slowness isn't on our end -- our importer would usually chug through the queue we've got now in a matter of hours, tops. The data we're receiving from LJ is just so throttled due to the DDoS that it's taking forever to chew through the queue!

[keiliss]

Aww, thanks for explaining things. I keep my DW account separate from livejournal, but with what's happening I finally cracked and created a new one to import my LJ into. I was just too slow *g* - been in the queue for 18 hours now. I guess I'll just have to see if it succeeds.

Me being offline wouldn't affect the import, would it?

[denise]

Nope! Once you start the job, it goes into the queue on our servers, and you don't have to do anything else -- you can close the tab, close the browser, turn off your internet connection, shut down your computer, seal your computer in a safe and drop it in a trench in the ocean, anything. :)

[florida_minxie]

I was importing from my IJ... is there anyway for me to stop it so that it won't take up the space in your queue?

[denise]

Nope, there's no way to stop an import once it stops. Don't worry, though -- IJ isn't having any problem, so when your job comes up (in another three days at this rate, heh) it should complete just fine.

[falena]

Thanks for the update. You guys are generally so very good at keeping your userbase informed. And that's why I'm so so happy to give you my money. I can't wait to renew my account come next month. :)

DW unavailability

[indigojo]

I have noticed that Dreamwidth itself is often unavailable, and I get "can't access" errors for several minutes on the go, at least once a day. Is this part of the same problem?

Re: DW unavailability

[denise]

Hm. Our server monitoring tools don't show any unavailability or high network traffic on our end that might account for this, and this is the first time I've seen someone mention it. (This is not to say that you aren't experiencing the problem! Just that I don't see anything that might account for it on our end.) It's possible that there's some network congestion on the steps in between you and us, or that there's some sort of routing problem in between your ISP and our hosting provider.

The next time it's happening, try doing a traceroute to dreamwidth.org (let me know if you need instructions), then open up a Support request in the Account Payments category with the results of the traceroute once you can reach the site again. (Or, put the results in an email to accounts@dreamwidth.org -- if you do that, be sure to include your DW username, just for reference.) I know it's not a payment-related issue, but that's the one support category where I'm 100% guaranteed to see it. I'll look into it from there.

[shutterbug]

Thanks for all this info. I started an import a couple days ago and it took about a full day to finish (but that was expected, and I'm just happy it finished with--from what I can tell--no problems; just slow). Unfortunately for LJ, this finalized my decision to come here for good. I hope for the sake of everyone who's staying there that LJ gets things under control and can prevent more attacks.

Anyway, I definitely appreciate all this info. Thanks for letting us know. =)