Import and crosspost problems from LJ

[denise]

Due to a change LJ made yesterday afternoon in how they process cookies and logins, some aspects of our crossposter and importer will not work at the moment:

* If you have chosen to enter your LiveJournal password every time you crosspost an entry, the crossposts will currently fail. The solution to fix this is to save your LiveJournal password on Dreamwidth (that method uses a different codepath, which has not changed) temporarily, instead of entering it each time. When we've been able to fix our code to use LJ's new login protocol methods, the option to enter your password each time will work again and you can remove the saved password.

* If you are importing content from LiveJournal, and choose to import your comments as well as your entries, the process will fail. (This is because comment import uses a different login codepath than entry import, and the codepath the comment import uses is the one that has changed.) To work around this, uncheck the 'comments' box when you're importing. When we've been able to fix our code to use LJ's new login protocol methods, the option to import comments will work again, and you can run a new import with comments included at that point. (Your entries will not be duplicated if you import again, and the comments will be added to the correct entries.)

I'm really sorry about the inconvenience and hassle! We're working with the LJ code changes to see what changes we need to make on our end in order to make things work again, and as soon as we can program and test a fix, and confirm that LJ's changes are stable and will not be changing further, we will make sure the fix gets pushed live onto the site.

Keep an eye here on dw_maintenance for further details.

Comments

[vaxhacker]

So should I assume that the current importer queue backlog is part of this same problem? I thought I had set things to avoid the issues mentioned here, like importing comments, but it's been sitting waiting for login validation since yesterday.

If that's just all part of the same issue with LJ's code changes, I understand, but in case it's in any way a new datapoint worth mentioning, it looks like all imports are stuck, not just the ones with those particular features enabled.

[jumpuphigh]

Every time a ton of people migrate from LJ to Dreamwidth and try to back up their multiple years worth of entries, the importer gets backlogged. I am not staff and cannot say that this is what is happening now but that would be my best guess as to the cause. Give it a few days and retry.

[vaxhacker]

Yeah, that's understandable. I'm not sure how much of what the importer is doing is visible on the webpage, but it looks like it's just stuck as opposed to overly busy.

Personally, I use a client I wrote at home to constantly back up my LJ posts and cross-post them to Blogger (because I have a few friends who only use that). I could just point it at DW and shift it into reverse and just upload all my LJ entries into DW myself, maybe that would be better than stressing the importer.

And FWIW my client is having the same problem logging into LJ as well:

> ./runsync -v
Opened journal for 'VAXhacker' (ID XXXXXXX)
Last synchronized from LJ Mon Oct 24 01:37:26 2011
Last synchronized to Blogger Sun Oct 23 18:38:59 2011
Connected to https://www.livejournal.com/interface/xmlrpc
Server challenges: c0:XXXXXXXXXX:XXX:XX:XXXXXXXXXXXXXXXXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
We respond: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Generated session token: v2:XXXXXXXX:XXXXX:XXXXXXXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX//1
Session ID: XXXXX
*** Downloading LiveJournal content (vaxhacker) ***
Expiring session XXXXX
xmlrpclib.Fault: <Fault 101: 'Invalid password'>

So I can get past the challenge/response, get my session and everything, but once I start trying to get data it throws an exception.

Assuming DW has essentially the same API access that LJ offered, it shouldn't be too hard to point my tools in its direction and just make DW my main journal.

[denise]

Taking a backup of your LJ for your local use is a good idea, but I wouldn't suggest doing an import manually -- you'll lose all the special features of the importer like auto-mapping of keywords, translation of commenters' usernames into their OpenID so they can manage their comments in the futures, etc. Just be patient -- I know people want to get stuff out of LJ, this happens every time LJ has a problem, but LJ is not likely to shut down in the next 3 days :)

[vaxhacker]

True, I hadn't thought about the translation features. Ironically, I'm not doing this because of the latest round of problems. I've been sitting on this DW account for a long time, intending to move over someday (the only reason I haven't sooner is that my friends aren't over here yet).

So my LJ paid account is up for renewal, and I thought "Well, no time like now to just switch over and upgrade the DW account." Then all of this happened. Timing, as they say, is everything. :)

[denise]

Hee! Clearly, the universe is trying to tell you something.

Welcome to the dark side! We have cookies.

[vaxhacker]

Cookies! Excellent.

Not sure if this is the right place for the question, but when the importer runs, will postings retain the same ID they had on LJ? If they end up getting re-sequenced from 1 (filling in the gaps left by deleted entries, etc), then I'll need to account for that with my backup stuff so it doesn't get confused about which posting is which.

[denise]

They unfortunately do get new numbers, yes!

argh hit post too soon. THey get new #s because the export function doesn't keep data that there was a post there that was deleted, so the system just fills them in in order.

[vaxhacker]

Ok, thanks for the info. I'll compensate for that on my end.

[denise]

Yes, it does look like changes LJ has made are causing problems. We'll give it a day or two to calm down before trying to troubleshoot too hard, since they're likely to make additional changes.