(no subject)

[denise]

Update on yesterday's post about importing: we are continuing to carefully babysit the importer to ensure the maximum number of people's import attempts succeed, and we'll continue to add more resources as necessary in order to keep it from affecting usage for the rest of the site as much as possible. Thank you all very much for your patience!

If you've been waiting for a slower moment to retry an import attempt that failed: now is a good time to retry, since the overall queue is pretty short! If you started an import job earlier today or yesterday and you haven't gotten a failure in your inbox yet: there are a few comment import jobs that we're trying to baby along because they involve a very very high number of comments or because the comment threads in the account tend to be very deeply nested or involve very complex response trees. Because we preserve threading data on import, in order to keep the imported entry looking as much like the original entry as possible, it takes a lot of working memory to do a comment import for large accounts or accounts with very complex comment threads. We've been tweaking the settings on how much memory an individual import worker is allowed to take up: usually all that memory management is handled automatically, but we're trying to find the "sweet spot" for allowing complex comment imports to succeed without running out of memory, taking up all the memory and leaving none for any of the other import workers running, or losing the safeguards for not letting a single process get stuck and just eat up more and more memory over time. alierak has been putting in heroic levels of work on managing the importer queue these past few days: thanks, Robby!

And, of course, some jobs are still failing because LJ is still intermittently blocking us, allegedly because "too many password failures makes a block happen automatically because it looks like someone trying to break into accounts". So there's still a chance your import will fail because it was assigned to a worker operating on an IP that's currently blocked. We're doing everything we can to work around that issue, but there's only so much we can mitigate it. If your import fails, wait half an hour and try again. (So far, the unluckiest person I've seen who kept hitting blocked workers took seven tries for it to finally work, but it did finally work!)

For people asking about Scrapbook photos: unfortunately, LJ doesn't provide a feed that would let us import those. lannamichaels has a method to at least download your photos, and blue_ant reminded me that Semagic, the Windows-based LJ client, will also download your photos (and notes that you may need to keep trying a few times).

Finally, because I've seen a bunch of people making references to deleting their LiveJournal account: before you delete your LJ account, it's a good idea to claim your LiveJournal OpenID with your Dreamwidth account. Doing this will cause all of your imported comments (including in communities), and any entries you made to a community that's imported, to update to having been made by your DW account instead of your LiveJournal OpenID account. Doing this before you delete your LiveJournal account will let you keep managing any old comments in your journal, any comments in a community that's imported, and any entries you made in a community that's imported, exactly as though you'd posted them with your DW account, and avoid the need to authenticate against your LJ account, which you can't do after it's been deleted. If you've already deleted your LJ account, we strongly recommend temporarily undeleting it and following that process before you delete it again!

EDIT: I forgot a lot of people don't know what OpenID is, sorry! OpenID is one of the protocols that lets you use one site's login credentials to log onto another site without having to create a whole separate account. If you've ever wanted to buy something from a website once, didn't want to create a whole account for it, and instead made the purchase with your Facebook/Google/AppleID login: that's the same concept. (Not exactly the same protocol, but the same concept.)

When the importer imports comments (or entries in communities), it attributes all the comments to the commenter's LiveJournal OpenID. That way, the person who left the comment can log into Dreamwidth using their LJ account and still have the same level of control they had over the comment (or community entry) as they had on the LiveJournal version of the entry.

Claiming your LJ OpenID account with your DW account means that when other people (or communities) import their journals, you'll be able to control those comments with your DW account instead of having to log in using your LiveJournal OpenID -- which you can't do anymore once you delete your LiveJournal account. It makes sure you don't accidentally lose control over comments and entries that you left in other accounts if those accounts have already been/are later imported to DW.

Comments

[primwood]

Thank you so much! This is a little confusing to me, especially about claiming my LJ OpenID. I have no idea what that is.

This morning I attempted to import my LJ. I have a zillion :) LJ entries. Apparently, it imported all my entries, but after 4 attempts, it could not import the comments. Do I have to do the whole process again, or do I just try to import the comments again?

Here was the last error message:

Your comments have failed to import.
Failure running job: Lost connection to MySQL server during query at /dw/cgi-bin/DW/Worker/ContentImporter/Local/Comments.pm line 173.

[denise]

OpenID is a protocol that lets you use one site's credentials to log onto another site without having to create an account -- if you've ever used your Facebook/Google/AppleID account to log into another site instead of creating an account, that's the same general concept (although those sites don't use the exact same protocol). When we import stuff, we attribute the comments to the commenter's LiveJournal OpenID so they can log into DW using their LJ OpenID, without having to create an account on DW, and have the same level of control over their comment they have on the LJ version of the entry. Claiming your LJ OpenID with your DW account means that, for instance, if your best LJ friend imports/has imported their LJ account to DW, you can control all of the comments you left for them back in 2005 on Dreamwidth, using your DW account, without having to log into DW with your LiveJournal OpenID. The reason to do this before you delete your LJ account is because once you delete your LJ account, we can't authenticate that you're the person who owned it, so there's no way for you to log into DW with your LJ OpenID anymore. Does that make more sense?

That error message means you hit one of the windows where LJ was blocking us: wait half an hour, try again. It may take a few attempts to succeed.

[beccaelizabeth]

I dont think I understand what OpenID is.

If you claim your livejournal comments can being logged in to livejournal somehow edit your dreamwidth?

If you delete your livejournal without claiming your openid and someone else claims your old livejournal name, can they then claim the openid comments with that livejournal name?

If you claim your livejournal comments openid and later someone else gets the livejournal name can that... do anything on dreamwidth?

[denise]

I edited the post to include the explanation I provided above! OpenID is one of the protocols that lets you log into a site using your account from another site instead of having to create a new account.

Claiming your LiveJournal OpenID account with your DW account will not mean that being logged into LJ will do anything to your DW. I believe LJ still prevents accounts that someone renamed to from using OpenID, but I'm not 100% sure: if they have changed that, there is a small chance someone who renamed to your old LJ username could claim your LJ OpenID on DW, which is another reason to do it now before you delete your account. If you claim your LJ OpenID account with your DW account, and someone later renames their LJ to your former LJ username, there's nothing they can do to your DW account with it. All that claiming your LJ OpenID with your DW account will do is update imported comments and imported community entries so that they're owned by your DW account so you won't have to authenticate against LiveJournal anymore, which you can't do once you've deleted your account!

[princessofgeeks]

Thank you all so much. I imported my LJ years ago but all this help you are giving now is so generous and wonderful.

[naomikritzer]

I already imported my LJ (years ago) and I'm actually not planning to delete it (I don't use it anymore, but Google search works better there than here for reasons I've never been able to determine). I started the process to claim my openID and hit this warning:

>>>Claiming this account will merge all of the comments and entries that are currently owned by the OpenID account into your account. This action is immediate and irreversible -- we will not be able to unmerge them.<<<

What does this actually mean?? I have no idea what this is saying it will do.

[denise]

Say you left a comment on your best friend's journal in 2005. Bestie has since imported their journal to DW. If you don't claim your LJ OpenID with your DW account, you would need to log into DW using your LJ account in order to manage that comment, which you can't do anymore once you delete your account. Claiming your LJ OpenID with your DW account means that comment in Bestie's journal will look like it was made by naomikritzer instead of naomikritzer.livejournal.com, you'll get emailed notifications if someone replies to it, and you'll be able to edit it or delete it without having to log into your LJ account and then use that LJ account to log into DW (which you can't do anymore once your LJ account is deleted).

The warning is just making sure you really really want us to update every imported comment/entry that looks like it comes from naomikritzer.livejournal.com to look like it comes from naomikritzer instead! As the message says, there's no way we can undo it, so we just triple-check.

If you want a better example: check out my initial announcement that DW was going to be a thing. That entry was originally posted to LJ, and imported to DW. The first comment thread underneath the post, those comments belong to zorkian and synecdochic: both Mark and I claimed our LJ OpenID accounts, so even though we originally made those comments on LJ, they look like they were made by our DW accounts (and we can now manage them with our DW accounts). The second comment thread was started by svmadelyn, who hasn't claimed their LJ OpenID with their DW account, so it still displays as svmadelyn.livejournal.com and they would have to first log into their LJ account, then use that LJ account to log into DW, in order to manage the comment.

[dawna]

I was able to get everything imported. It took a few days, whatever you did last night was the sweet spot for the comments. It said they failed (like the bunch of other times), but they are all there, found out this morning when I came back to try again.

Thank you for working on this.

[michelel72]

I had a quick, clean import today (though it's possible that my old, original import already handled everything and there wasn't actually anything left to bring over).

Some suggestions that might help other folks:
- I logged into LiveJournal first, and they made me change my password because it was "too old", so that could be a failure point for some folks.
- Another could be use of the Google password manager, which might be defaulting folks' DW password in the import-kickoff pages; I just made sure to paste the new LJ one at each checkpoint.

[denise]

Oh, good call, I forgot that LJ was doing the "change your password" thing. (Hilariously, except it isn't really hilarious at all, I just have to laugh because otherwise I'll scream: it's not because your password is 'too old', it's because your password was compromised in the massive LiveJournal security breach they have never admitted happened, much less disclosed to their users. If you used that same password anywhere else on the internet, for any other account, anywhere, go change it there, too: the dumpfile is now widely circulating and it's trivially easy for someone to get a copy of it.)

[katuah]

I don't think I ever set up my LJ to be used as an OpenID account - would they have made it one automatically?

I imported my LJ to DW _years_ ago now, and rarely ever go back to LJ. I don't want anything about my DW to change at this point. If we don't intend to delete our LJs, is there really any point in doing this claiming thing?

[denise]

You don't need to 'set up' anything for your LiveJournal account to function using OpenID. It's exactly like if you want to buy something from a site that you know you don't intend to return to so you don't want to create an account for that site and just log in using your Google account instead! When we import comments (and community entries) from LiveJournal, we show them as having come from LiveJournal, so people can still manage them with their LiveJournal accounts (by logging into DW with their LJ account).

The reason to claim your LJ OpenID with your DW account would be to make sure you can still manage those imported comments and community entries without having to log into LJ. For example, if you made a post to a community on LJ in 2005, and that community imports all its entries to DW, the DW copy of the post would look like it was made by katuah.livejournal.com (or whatever your LJ username was). If you wanted to edit or delete it, you'd have to log into LJ, and then use your LJ to log into DW. If you claim your LJ OpenID with your DW account, that entry would look like it was posted by katuah, and you'd be able to edit or delete it with your DW account.

I know this is a bit confusing, so an example: here is the post I made to my LJ account first announcing Dreamwidth. That post was made on LJ, and I imported it to DW. The first comment thread on the post shows as having been made by zorkian and synecodochic: both Mark and I made those comments on LJ, using our LJ accounts. We both claimed our LJ OpenID accounts with our DW accounts, so now we can manage those comments while logged into our DW accounts if we want to. The second comment thread was made by svmadelyn. They haven't claimed their LJ OpenID with their DW account, so it shows as having been made by svmadelyn.livejournal.com. If they wanted to manage that comment, they'd have to first log into their LJ account and then use their LJ account to log into DW in order to manage it.

If you don't intend to delete your LJ, the only reason to do it would be convenience! But it's a good idea anyway, especially if you participated heavily in communities on LJ, just to make sure you'll always be able to control the imported DW copy of comments/community entries that you left ages ago.

[snacky]

Hi Denise! Thanks for this update, especially on OpenID!

I have a question. Is it possible to import a community that I'm a moderator and maintainer of on LJ, but not the owner? I tried, but it asked me for a password. I'm not sure if it meant my snacky LJ password, or the password for the community, which I don't know and am not sure if it ever had one.

[denise]

I am not 100% confident in this answer, because I haven't kept up on any changes LJ made to how comms work, but I'm 99% sure that any maintainer (but not any moderator) of a community can import it. The password it's asking for is the password to the LJ account that you told it is a moderator of the community.

[prettyarbitrary]

Oh dang, I did not know OpenID let you hang onto posts that way! (Semi)tragically, I imported years ago, so it's way too late for me to go back and claim my LJ OpenID, but I love it anyway.

[denise]

Yeah, sadly, if you deleted your LJ account long enough ago that it's been purged, there's no way to authenticate against it in order to claim your OpenID. Which is why I'm glad I remembered to warn people this time!

[squirrelitude]

For moderately tech-savvy people who are planning on deleting their LJ accounts, I have a script that will first go through and "burn" each post and comment by replacing its content with junk. *Then* you'd want to delete the account. This is potentially more thorough than simply deleting. (It does not remove your content from their backups, of course, but it prevents them from merely soft-deleting your journal and posts.)

This is obviously a pretty dangerous tool, so use it carefully: https://github.com/timmc/lj-expunge

(I haven't tested it in a while. I assume it still works, but no promises, and I won't be able to offer support. The script is offered as-is.)

[silveradept]

For people who are considering using this tool, it's written in Python 2, so you'll need to run it in a Python 2 environment. (Python 3 did a lot of breaking changes that would take significant rewriting of the script to make it work.)

[meowmensteen]

Wow, thanks alierak for keeping the importer queue up and running. It worked for me on my second try.

[cassiope25]

Hi, I just wanted to go through with your advice about the OpenID but I think I got it wrong as I tried ad logged in here on DW with my lj account. I think it should be the other way round but didn't find the possibility to log in with OpenID on lj.
So, what does "claiming" my lj account means? What do I have to do?
Thanks for the help :)

[denise]

Let me give you step by step instructions and see if that helps!

1. Log into your DW account cassiope25 on Dreamwidth.
2. Log into your LJ account in another browser tab.
3. Once you're logged into both accounts, go to the Dreamwidth tab, and go to https://www.dreamwidth.org/openid/claim
4. In the box on that page, put the URL for your LJ journal. (For instance, if your usernames are the same on both sites, you'd put https://cassiope25.livejournal.com in that box. If your usernames are different, replace 'cassiope25' with whatever your LJ username is.)
5. Click "claim account". You will be forwarded to a LJ page where it asks you to allow the authorization: say yes. (I don't remember the exact wording and I don't have a spare LJ account to go get the exact wording, sorry!)
6. Once you allow the authorization, you'll be returned to DW and it will ask you to confirm that's what you really want to do; say yes.
7. After an hour or so, you'll get an email asking you to confirm again that's what you really want to do. Make sure you're still logged into the right DW account, and click that link.

[alexeybobkov]

A very nice idea about converting OpenID to Dreamwidth Id, thanks!

[weatheringtea]

Hi Denise, thank you so much for everything you've been doing with this!!

I'm the owner of a 20yo chonker of an LJ community that I've been waffling on transferring over and archiving on DW. What I'm worried about is the size of the community and the strain it'll put on your resources (and if it's even feasible). Is there someone I can I reach out to about getting that done safely (I guess is the best way to put it), so I don't screw things up for your importer and everyone else trying to bring in their own journals?

[denise]

Right now we're reasonably confident that we've tweaked enough stuff that even very large imports should succeed! Give it a try, and if it fails, let me know and I'll check to see if the failure was anything other than "blocked by LJ at that exact moment".

(To put "very large" into context: we have a comment import with 7 million comments in the queue right now and we're about, like, 85% sure it's going to work.)

[tawdry]

I was going to suggest LJ-Sec to copy entries over, but it looks like the dev let his domain expire and all the versions were hosted there. That's what I used when I left LJ in 2014. Do they allow an XML export/download of entries that could be uploaded into WordPress or something so they'll still be accessible if the importer fails?

[denise]

We don't allow people to import from file (because it means they could edit other people's comments) but LJ does let you export month by month as xml. Don't give up on the importer yet, though! We are keeping it going as much as we can, and even if it takes people a few tries to get it at a time when our import workers aren't blocked, many are succeeding.

[voidampersand]

Thank you, denise. I decided DW was the right place for me, years ago, and yeah, it still is.

[denise]

I'm so genuinely thrilled we could make a place that feels like home to you, since that's precisely what our goals were all along <3

[spatzenpost]

Thank you so much! I was wondering how that worked, and it turned out perfect!

[tochira]

Thank you all for your hard work, a million times over. And extra thanks for mentioning the importance of managing OpenID, because I had totally forgotten how it is set up - I did it so long ago!

Newbie! Howdy!

[seeking_ataraxia]

Hi! So most of my stuff on Livejournal was Friends only. When I import, will it now be public???

Re: Newbie! Howdy!

[denise]

No, all your entries will import at the same security level they had on LJ!

[afra_schatz]

Hello!

First of all, thank you so much for all your work and information. It's much appreciated.

Sorry if the question I have is one you already answered, I am a little confused and not sure, so I thought I'd ask:

I have mirrored my LJ to DW years ago and since then, no comments have been made to my LJ that I want imported. Can you tell me whether I should still temporarily un-delete my LJ to follow the steps of claiming the OpenID as you suggested in your post or is that unnecessary?

EDIT: I read in another thread in this post that it might be important to do so, so that in future, should someone set up an LJ account with the name of my (purged) former LJ name, they won't be potentially able to have access/could edit something in my DW account - that is of course something I don't want :). But perhaps I misunderstood that thread, sorry if I sound a little bit confused an all over the place :).

Thank you so much in advance for your help!

[denise]

No, if your LJ account is deleted and purged and someone renames to the same username, they won't be able to manage your old imported DW comments! But it's still a good idea to claim your LJ OpenID: unless you claim it, you will lose the ability to manage, on Dreamwidth, comments that you left on LJ that were imported to DW and community entries you made on LJ that the community admin imported to DW.

In other words: claim your LJ OpenID now before your LJ account is deleted and purged so that you keep the same amount of control over imported-to-Dreamwidth comments left in other journals and community entries as you had when you originally made them on LJ.

[med_cat]

Thank you for the detailed info, as always, Denise!

EDIT: Please disregard my question below; my OpenID works!

:)

Question re: claiming OpenID:

"Right now, if your username on the remote site has an underscore in it, you might not be able to claim your OpenID properly. This is a bug we're working to fix."

Has this bug ever been fixed?

Many thanks!

[denise]

Yes, we worked out a fix for everything but when the underscore is at the beginning or end of the username!

[daisychains1957]

Thank you so much for all your help, I have now claimed my OpenID from LJ :)

[violateraindrop]

That is awesome! A lot of communities I was/am active in are moving over. This should make the whole process even more seamless. Seriously, thank you so much for all your work making this easier for us LJ users ♥

[squirrelitude]

Here's a post about exporting Scrapbook images: https://siderea.dreamwidth.org/1754427.html (currently gives two strategies, second one sounds easier)

This doesn't help with getting them into Dreamwidth, but it does give you a way to get them downloaded from Livejournal.

[rifleman_lincoln]

Thank you for putting in heroic levels of work on managing the importer queue... It's really appreciated.

I'll have another go!