Friday import update

[denise]

We have managed to chug through every single outstanding "things broke because there were too many comments/something in the data feed was busted/we have no idea why it broke but it runs okay when we run it manually" import that people alerted us to, and the import queue has been holding steadily at 0 for the last 24 hours (ie, jobs are finishing pretty much as soon as they're scheduled), so we've cut back the extra resources we diverted to the importer for this week. If you noticed a brief blip (like 30 seconds) of the site giving 502 errors, that was us fiddling with the memory allocation and the databases again!

We thought we'd figured out the underlying cause of why imports with a large number of comments were failing, but every time we've thought we solved the thing that would fix it, it's turned out the actual issue is down one layer of abstraction. We're still working on figuring that one out. Yes, it's driving us up the wall. We're sorry! We will let you know when people can do large imports again. (If it turns out after we find and fix whatever bug is causing the problem that we also need extra memory for those to succeed, we'll throw that extra memory at it for another few days at that point so people can have another window to import their larger accounts without the job running out of memory -- we just have to turn it back down for a while because it is more expensive to provision that much memory and we don't have the budget to do so indefinitely, especially if it's not being used like it isn't right now. But a few days is not a problem.)

Thank you again to everyone who's been so patient with us this week. We really, really appreciate it.

Comments

[evalerie]

And thank *you*, Denise, for being there keeping DreamWidth running!

[pantswarrior]

I've just been watching this stuff happen since I didn't have anything massive to import (I was reminded of a tiny community, it went through within a minute or so last weekend), and being incredibly impressed. In short, I must assert that: you kick butt. :D

Thank you

[akatsija]

Thank you very much for the update. It is really appreciated.

--
А.Кац и Я

Layers

[frith]

Now we know, on what rests Dreamwidth island as it sails through the ether. A layer of abstraction. And on what rests that abstraction? Another layer of abstraction. And under that? A layer of abstraction. And un... Abstractions! It's abstractions all the way down.

Re: Layers

[batrachian]

Until you hit the bare metal (or silicon), anyway...

Re: Layers

[satsuma]

"abstraction on top of rocks that think" sure is a cosmogony 😆

Re: Layers

[filialucis]

Now I wish there were a Like button for Dreamwidth comments. :)

[badly_knitted]

You guys are such troopers. Thank you!

[cf2princessawnw]

Can't tell you how much WE really REALLY appreciate your team and you here!!

[emrys_mk]

Thank you for all of your hard work! I have imported several journals and they’ve all gone through quickly.

[smutmods]

Thank you immensely for your help and dedication to solving this for us. You have a very loyal userbase and it's obvious from things like this just how well deserved that is. Thank you again. <3

[denise]

Our users really are amazingly lovely, aren't they?

And yeah, it's really nice to have finally proven I was right in those old arguments I used to have with the former CEO of Six Apart about how people would respond so much better to us if he'd just let me write a goddamn fucking news post in which I sounded like an actual human being as opposed to a robot that had been programmed to spew marketing talking points. I mean, it's a total Pyrrhic victory, and he'll never know I was right, but I was right, goddammit.

[archangelbeth]

(You were SO right.)

[kaishin108]

Thank you for all your hard work! :-)

[darkoshi]

Thank you all for your efforts and for being so diligent!

[subsequent]

I haven't been paying too close attention to this (primarily because I imported my content years ago) - but I just wanted to echo the thanks in here. The care that the people put into this place is evident, and it's nice being part of a community that has this level of love ♥

[winter]

Mine went through manually at 300K+ of comments nested 1000 deep, and you are superstars of responsive support! Thank you so much for everything you're doing for us!

[freedom_of_sea]

Would you make Chrome extension (client-side) to sync everything ?
Decentralising things rocks!

[denise]

We don't have anyone who's good at that right now, sadly! And I strongly suspect LiveJournal is gearing up to remove all third party client access, although that's just a suspicion.

[pangolin20]

Although I have not had anything to do with this, I still want to thank you for the immense effort you are putting in and for not giving up. You are truly dedicated and I am glad to have you running this website.

[torylltales]

I didn't have anything to import (I flounced from LJ years ago and deleted everything before I knew there was an importer here), but I just wanted to say thank you to the development team. The care and attention you give to the users of this site, and your dedication to improving and maintaining site functionality and user experience overall, has been so refreshing compared to some other social media sites. I really appreciate all the hard work you and everyone has put into DreamWidth.

[violateraindrop]

Thank you very much for putting so much work into this ♥

[havocthecat]

I just appreciate all the work that you're going to with all this, especially given, uh, how I imported one of my more important-to-me but defunct roleplay journals over here during the whole importer break without even realizing you were working on it. (I just kept retrying it and clearly something you were doing worked, it got farther and farther in the process each time!)

[denise]

That's hysterical. I'm glad it worked! It's been a while since we've had a good stress-test of the importer like this, heh.

[wantedonvoyage]

My epic import finally worked after a few tries. Thank you so much for all you are doing!

[ulrith]

Please accept my great appreciation for your work!
I made few attempts of import of my journal from LJ in the past but they all failed.
But today I've finally done it and closed my account there.
I'm considering paid account in DW now.

[chocolamousse]

I imported my LJ years ago but I've read all your recent posts and I'm really impressed by your commitment, the way you update us, your efficiency and your general kindness. Many thanks to you and Mark for your hard work and for making Dreamwidth the great thing that it is. :-)

[rifleman_lincoln]

*applauds your dedication*

[kadenza]

I moved to DW in 2017. A few days ago I suggested that it was time for my sister to make the switch as well. Moments later she reported that it had come through just fine. Amazing! I appreciate all your hard work.

Question for the community.

[vinyl1]

Hi. I have just discovered Dreamwidth in the course of searching for a home for my community that is now on Live Journal. A lot of the group is very uncomfortable with a Russian-based server right now.
Our archive at Live Journal is huge. We have 50-60 posts a month with lots of comments, and we go back to 2006. Does the community think an import that large would be feasible?
My other question concerns the posters and commenters. Since they would not have an ID on Dreamwidth, how would the entries and the comments come out?
Thanks for any help you can give.

Re: Question for the community.

[denise]

1. If 'a lot of comments' means more than 300,000 Comments Received (check on the profile for that), you'll need to start the import once, have it fail, and then email us at support@dreamwidth.org to have us run the job manually. (It's very important to run the job once even though it's going to fail, or else we won't be able to run it manually!) Our ability to run imports manually may disappear at any time and without warning, because LiveJournal is continuing to block our servers as they identify them and we're getting close to running out of available unblocked servers, so if you're at all considering it, consider it quickly.

2. Comments and community entries are imported as attributed to the original poster's LiveJournal account using OpenID, a protocol that lets you log into one site using another site's account. People who posted them can log into DW with their LiveJournal URL and continue to manage the content, or they can create a Dreamwidth account and "claim" their LiveJournal URL to have the comments and entries update to belong to the Dreamwidth account instead. I explained it a bit more at the bottom of this entry.

Re: Question for the community.

[vinyl1]

Thank you so much - that definitely sounds promising!
The archive would definitely be nice to have, since LJ may disappear at any time.

Thank you!

[deacertes]

Thank you for all your hard work with this. This site has lifted my spirits during difficult times and I very much appreciate it and you.

[kriscynical_2022archive]

Can you give an update as to whether comment importing has been fixed on large journals, pretty please? When I imported my old LJ account a month ago, everything imported except the comments. Since my journal from back then IS so large, I don't want to try importing everything again until I can (possibly) know whether or not the comment issue has been fixed.

That being said, thank you so much for working on the importing feature! The entirety of my college experience from 2003-2007 and the years immediately following are on that old LJ account, and the entries and conversations I had with my friends are very dear to me. I'm so thankful that I've at least been able to archive the entries themselves so far!

[denise]

We fixed about half of the problems we were having, so give it a try and if it doesn't work, let us know and we can manually run it for you if you're one of the other half

[kriscynical_2022archive]

Is there a way for me to import JUST the comments, or do I need to do the entire process over again? If I need to re-import everything, will that cause double entries? Or do I need to wipe this account's content and start over from scratch?

Thank you so much for responding; I really appreciate it.

[denise]

You can't import comments without entries, no: otherwise the importer could have comments without entries they belong to and bad things would happen. Running another import does not create duplicate entries, only fills in any missing ones.

[kriscynical_2022archive]

Everything imported successfully, including the comments! Thank you so much for your help; it's such a relief to finally have a full archive of my LJ. :)

[tinny]

Thank you for all you've done about the importer issues! You rock!

I have a question about claiming OpenID. I have imported my comms to DW and I can see that most people have claimed their OpenIDs (including me), but the ones that haven't all are people with underscores in their names.

The FAQ for claiming OpenID says that underscores might be a problem, but the FAQ post is from 2015, so I'm not sure if it's still accurate. I didn't want to open a support request for this either, since I'm not actually personally affected.

So: is this still a problem, and could you maybe think about fixing it so that my comm members with "unfortunate" names can also claim their OpenIDs?

[denise]

It should work for most people with underscores in the middle of their name; it won't work for anyone with underscores at the beginning or the end. We banged our head against how to fix it for a really long time, but it's a fundamental flaw in the OpenID protocol and fixing it 100% would require LiveJournal's cooperation, which is basically never going to happen at this point (we can't get them to even reply to us about why they're blocking all our IPs, etc).

[tinny]

All right! Thank you so much for your fast answer!

I assume you could probably manually set the claim for someone if they asked support (and could prove they have the right to claim it), but I'll leave that up to them.

[denise]

We don't have a secure enough procedure for "proving they have the right to claim it", unfortunately

[byslantedlight]

Hi - hoping you're still monitoring these comments! First of all, thanks for adding the extra resources a while back to cope with all the extra importing. :)

Running another import does not create duplicate entries, only fills in any missing ones.
You said this in one of your comments above, and it made me wonder - I'd like to update an import of a comm that I run, but I don't want to risk losing the successful import from a few weeks ago. Your comment makes it look like the old import wouldn't be lost though - it would just add the new entries. Is that right, or am I just being hopeful?!

[denise]

You are correct, it will just add the new entries!