LiveJournal slowdowns/DDoS affecting importer, crossposts

[denise]

Importing

As I'm sure many people have noticed, the waiting period for our content importer is incredibly long right now. Usually, an import will take minutes; right now, the queue is probably at least 24 hours long, and is only getting longer, not shorter.

This is because most of the import jobs waiting in the queue are requesting data from LiveJournal. LiveJournal has been experiencing a Distributed Denial of Service (DDoS) attack on and off since March 24. This means that the site is often unreachable, either because of the network traffic brought about by the DDoS or because of tactics being taken to block the network traffic brought about by the DDoS. (In other words, sometimes being unable to reach LJ is because of the clogged network, and sometimes it's because of a 'false positive' identification as part of the problem and a temporary block.)

These temporary blocks and timeouts when accessing LJ affect the importer (and the crossposter!) as well. The importer and the crossposter both try multiple times before giving up, and each of those tries takes time. Sometimes an import job will get through, because it hits LJ at a time when traffic is lower. More likely, the import job will loop through its scheduled attempts, then fail.

What this means for you:

* Trying to import your journal right now is extremely delayed, and will likely only get worse. Please consider waiting a few days or a week to try to import your journal from elsewhere.

* If you do start an import job, it will take a very long time to reach the front of the queue. You will not receive any results from your import for at least 24 hours or so. It's okay to close the window or tab where you started the import. You'll receive a notice in your Dreamwidth Inbox when the job runs.

* It is very, very likely that your import job will fail. If your job reaches the top of the queue at a time when LJ is unreachable, the job will retry a few times, then error out. Please consider waiting a few days or a week to try again.

* Each individual step of the import runs under its own separate job. This means that some of your data may import while other bits won't. When the problem calms down, you can re-run your import and only bring in the data that it missed. (You can run an import as many times as you want. Each time will only pick up new content; it won't result in duplicates.)

* Edit, because people have asked: There's no way to cancel an import job once you start it. It's all right, though -- you can close the tab/window and log off, and the process will just stay in the queue until it's your turn.


Crossposting

Crossposts are also occasionally affected by LiveJournal being unreachable. If you try to crosspost to LiveJournal and the job fails, you'll receive a notice in your Dreamwidth Inbox (it should say something about the XML-RPC interface being unavailable). Once the problem calms down, you can "retroactively" crosspost the entry by using the Edit Entry link, making sure the crosspost option's box is ticked off, and save the entry. You don't need to make any changes to the entry itself.


Our fingers are crossed for LJ that they can overcome this problem soon! (Hopefully before their operations team has a nervous breakdown.)

Comments

[denise]

It will, yes. the importer and crossposter talk to each other :)

[a_muse_d]

simply, thank you for the update

[highlyeccentric]

Oh, that is shiny. :D

[faithofone]

*that* is cool

I'm imagining two guys sitting around talking about their jobs and being all "did you see what so and so just imported?"

"No, but I saw what they were crossposting. Ohhh My!"... oh darn now that guy is George Takei in my head.

[elialshadowpine]

Thank you for keeping us updated. :)

I also want to say, you guys have been hella classy dealing with all this. I've seen a lot of blame going toward LJ for the DDoS, although mostly from people who are not techies, and it's really nice to see you guys supporting them even though they are technically your competitors. My husband works as a sysadmin, and I know how stressy DDoSes have been for him at his company. The LJ team has to be going nuts by now. So thank you again. You guys are awesome.

[discodiva76]

Well I must have been incredibly lucky....I started my import a couple of day ago and decided to stay in the queue and see what happened (am going away in 5 days and wanted peace of mind before I went, just in case I came back and found LJ gone for good!)....

My entries successfully imported 3 hours ago.....just waiting in the queue for the comments now...

Thank you DW and LJ....I am very relieved to have backed up my entries at least.....I can head away knowing that 6 years of mindless rambling is safe for a while..

Deeds xx

[vlredreign]

This explains a lot. Thanks for the info.

Question: has this been happening when trying to import from InsaneJournal, or does it not matter where the import comes from because of all the LJ imports being backed up? I want to import my posts from there as well, but I'm gonna wait awhile, I think.

[denise]

It doesn't matter where -- imports go into the queue in order no matter what the source, so your IJ import would be behind the 1127 (at this minute) entries imports from LJ. :)

[samjohnsson]

... oh darn now that guy is George Takei in my head.

...and suddenly, I wish DW had a "Like" button.

[samjohnsson]

Not an issue affecting me overmuch, but thank you for keeping us in the loop as to what's going on!

[etfb]

The use of the inbox for error messages is problematical, to put it politely. When you crosspost, you get a message in your inbox. When you cross post every time you post, your inbox fills up. I write a blog post every day, and have done for several years now, so my inbox is full of these redundant messages and as a result is completely and utterly useless for anything else. I'm effectivaly spamming myself, because DW doesn't give an option to switch off those successful messages. Until that's fixed, I might as well assume that DreamWidth doesn't provide any kind of inbox service. That's a pain.

[vlredreign]

ZOIKS!!!!

That's what I feared. Thanks again!

[denise]

You can filter to entries, comments, and private messages to view only those items. We have a bug open to allow users to choose whether or not to be notified on successful crosspost (notifications on failure will always be mandatory), but the notification system is a maze of twisty passages all alike and we need to pay some technical debt before we can change it.

[dimrub]

Thanks for the heads up! I like the way you guys keep the folks up to date. A partially unrelated question, if I may. This latest wave of troubles over in the LJ land caused something of an exodus, and dw seems to be the primary destination for those abandoning ship. What is your policy of giving invite codes to the newcomers? I'd like to invite many of my LJ friends over, but not all of them are comfortable using their credit card to pay for a paid account (a paypal support would have been most welcome).

[denise]

We give out invite codes to our userbase when we feel we have the resources to support more accounts! It usually tends to be once every month or two. We use different criteria to distribute the codes, too -- sometimes we give codes to paid account holders, sometimes to active users, sometimes to people who have used all their invites, etc.

Believe me, we'd love to have PayPal support too. PayPal dropped us last year after an organized troll group created a fictitious concerned-parent organization and reported content on DW to PayPal. PayPal decided that the content -- though perfectly legal and not against our Terms of Service -- freaked them out, so they demanded we remove it as a condition of remaining a customer of theirs. We refused, and PayPal closed our account with them. Payments were totally down for about three months, as a matter of fact, until we could implement our new payment system -- we couldn't take in money except via check/money order during that time.

PayPal is never going to reinstate our account unless we comply with their demands, which we're not going to do, so your friends can check out dw_codesharing, where people share invites they have available. There's also the option of having someone who does feel comfortable using their credit card buy a paid account creation code for them and they reimburse that person through PayPal (or wherever)! We also accept check and money order in US funds, and if cash gets to us we'll apply it to an order (we can't guarantee that it won't get lost in the mail, and there's no recourse if it does, so this is best used for small orders).

[dimrub]

Too bad about PayPal - and kudos to you guys for standing on your own. And thank you for such a detailed response! The trick of paying for someone and getting reimbursed is definitely something to consider.

[tree]

maybe i'm missing something really obvious, but why don't you just delete the confirmation messages from your inbox?

[faithofone]

Awee, I feel special :D

[goodbyemyfancy]

Thank you so much for the information! I'm trying to import from LJ and it's been 2 days now, so I think I'll try to stop it and redo it in a couple of weeks. Wanted to use my account here as a back up to the insanity over LJ but when everyone does it at the same time, it's certainly changes things for DW!

[denise]

You can't cancel an import once it stops, but it will either finish or abort eventually!

And the slowness isn't on our end -- our importer would usually chug through the queue we've got now in a matter of hours, tops. The data we're receiving from LJ is just so throttled due to the DDoS that it's taking forever to chew through the queue!

[keiliss]

Aww, thanks for explaining things. I keep my DW account separate from livejournal, but with what's happening I finally cracked and created a new one to import my LJ into. I was just too slow *g* - been in the queue for 18 hours now. I guess I'll just have to see if it succeeds.

Me being offline wouldn't affect the import, would it?

[goodbyemyfancy]

WOW - thanks for the very fast reply on this! Too bad I cannot stop it, but I'll just it let run and see what happens :)

[denise]

Nope! Once you start the job, it goes into the queue on our servers, and you don't have to do anything else -- you can close the tab, close the browser, turn off your internet connection, shut down your computer, seal your computer in a safe and drop it in a trench in the ocean, anything. :)

[florida_minxie]

I was importing from my IJ... is there anyway for me to stop it so that it won't take up the space in your queue?

[denise]

Nope, there's no way to stop an import once it stops. Don't worry, though -- IJ isn't having any problem, so when your job comes up (in another three days at this rate, heh) it should complete just fine.