New imports temporarily stopped; import queue paused

[denise]

The situation with LJ's DDoS affecting our import queue has only gotten worse since my last post, and delays/slowdowns from LJ means that the queue will not clear anytime soon (due to the length of time each job takes to complete or fail with the extremely slow connections we are experiencing). At this point, it's also possible that the traffic from DW's importer is making LJ's situation worse, which we absolutely do not want to do!

We are therefore temporarily pausing the existing import jobs until the traffic clears up some and we are able to further modify the importer so that it is even less of a load on LJ. We are also temporarily stopping the ability to queue new imports, so that once we bring the import queue back up, it will give the queue the chance to clear a bit. (Also, we'd like to minimize the impact on LJ.)

We will re-enable the existing import jobs in the queue once we are able to make some changes to the importer in order to lower the risk that we're making things worse for LJ even more (reducing the number of times a job will try before it fails, further rate-limiting our queries). At that point, we will let the existing jobs in queue run for a while until we can clear out the queue a bit. Once the queue clears, we will re-enable the starting of new jobs.

It will be at least a few days until we are able to re-enable new imports.

If you already have an import job in the queue, you don't need to do anything. It will be automatically re-started when we bring importing back online. (It's not possible for you to cancel an import once it's started -- sorry!)

If you're importing your journal because you want a backup of your LiveJournal, you may want to try the LJ Archive software, which will give you a local copy of your journal. (You can't upload it into your DW account, but at least you will have the backup!)

If you're trying to import from a source other than LJ, I'm really sorry that the LJ DDoS slowdowns are affecting you! We're working right now on the ability to only pause imports from a single source, rather than all of them. Once we have that, we'll re-enable imports from sites other than LJ, and let the LJ imports sit in the queue until LJ is feeling better.

I'm really sorry about all the inconvenience!

EDIT, 2PM EDT 4/10/11: We have just pushed live a change that will allow us to re-enable the queuing of imports. Imports from sources other than LJ will now be processed as normal. Imports from LJ will be held in the queue and not processed until things calm down a bit on their end.

You can therefore submit an import request to import from LJ and forget about it. We will run the LJ imports once the DDoS and resulting issues clear up a bit on their end -- it could be a day or two, it could be next week; we just don't know!

Comments

[goodbyemyfancy]

Thanks for the update with this, and a very good call!!

BtW - LJ Archive used to be great for backing up our LJ journals. Unfortunately the software is out of date and no longer works as the author is no longer maintaining it. I used to use this all the time. I tried using this before setting up my import from LJ to here. Perhaps it works for others though.

[denise]

*nod* It's the one I hear about most often from people making recommendations! If anyone has a better option, let me know and I'll edit the post.

[shinsetsu]

I've used LJ Book to turn my DJ into a nice PDF. It worked wonderfully well. It may not now during the LJ attack, but I was very happy with the results, and it is free. There are more options if you want to actually print it as a book, however.

http://www.ljbook.com/

[goodbyemyfancy]

Cool - looks like a great service but I just tried to use it and got the message "Sorry, but too many users are currently using this service."

LOL! Seems that everything is swamped right now :) I'll keep it bookmarked and try again later :)

[denise]

It just goes to show that everybody knows they should be doing regular backups, but somehow manages to totally forget until something reminds them and then everyone wants to back up all at once. Which of course is usually that they're worried about the site they're backing up from, and the huge load of everyone trying to back up makes it look like something they should be worried about! Vicious cycle!

[goodbyemyfancy]

Just wanted to give you a personal thank you for your quick and insightful replies - 6 years on LJ and I've never had *any* replies to my posts to LJ news or tech questions - let anyone replies as fast as you make them to our posts!!

And I am so guilty of knowing I should do backups more regularly and failing on it :)

[denise]

Well, it is a lot easier to reply to people when your news posts only get 100 comments instead of 5000 comments. :) I'm sure that if we ever reach that traffic level, I will be just as burned out on answering comments and will pick and choose myself!

But you're welcome. :)

[faithofone]

If official posts start getting that many comments, hopefully there will be increased funds going Dw's way as well and you can hire more minions to help!

[ariandar]

Thanks to your icon, I now want gumdrops.

[faithofone]

:D I don't have any at the moment, those were from about six months ago. But I'm willing to share my Skittles if you'd like some.

[sunrisetr]

I have used LJ Archive just the other day and got a working archive (at least, as far as I can discern), but I couldn't import the comments. The software has been updated this February, as its Sourceforge page says.

[helens78]

The software is the same version number (0.9.7) it's been for years, though.

[goodbyemyfancy]

That is still the 2005 version of LJ Archive - it has not been updated but just reposted as far as I can tell. The comments say the same, though some people can get their entries for some reason - never the comments though.

[sunrisetr]

Meh, that's pretty underwhelming, then. Thanks for the heads up! There really seems to be very few backup options right now.

[brownbetty]

There's antennapedia's migration tool, which, although I last used it in 2008 or so, shows to have been updated in 2009. It's command-line, but has some decent tutorials on use linked.

(And I totally stole huge chunks of her code to make a thingy for myself that I have since broken, that reposted stuff from one of my blogs to another.)

[sunrisetr]

Thanks a lot, will definitely look into this! It's newer than LJ Archive, at least.

[yukinojou]

I've been using LJ Archive as recently as this week, and it creates a backup of both my personal journal and my RP community (with comment threads of 1000+ comments) without issue.

[pne]

Mine always tells me it downloads entries and comments. (I haven't looked at the results recently, though.)

I also use ghewgill's ljbackup Python script, which dumps each entry into a separate XML file and dumps comments into XML files (one per entry) as well.

I imagine that could be easier to post-process than some .NET serialisation format (which is what I think LjArchive uses) - since XML is just text, if push comes to shove, you can just open them in Notepad and copy the text out. (And if you're a programmer, you can do all sorts of fancy stuff with XSLT or something.)