(no subject)

[denise]

We believe we've figured out the underlying reason why larger import jobs (300,000+ comments) are failing! Right now we're manually running a few larger jobs to verify that our current theory is correct (it's a convergence of several different factors), and if we are correct, we'll put in a quick and dirty fix so those jobs will no longer need manual intervention. In the meantime, if your account or community has more than 300,000 comments, hold off on restarting another import: we'll let you know when you can try again.

Comments

[elwendell]

And this is why I love DW. <3

[chanter1944]

Exactly! <3 All the warm fuzzies to the DW staff.

[dotinthesky]

Good to hear. :-)

[rafiwinters]

I imported several years ago (successfully) but I am commenting here just to say thank you for all this hard work for us! :-)

[andrewducker]

Thank you for your work on this!

[svet_ka]

God bless for all your work and support!

[ceemage]

it's a convergence of several different factors

From my own experience in IT (ERP systems rather than journalling!), that always seems to be the way. Thanks for your persistence on this!

[denise]

After 13 years we've certainly found all the easy bugs.

[jamoche]

Oh, to have had a management chain that recognizes that there are no easy bugs left in complex code that's been in active development and optimization for 30+ years. Instead of, say, making a PIP task to find ways to improve it *without* looking at the existing bug database.

[denise]

Oh God I winced just thinking about it, my condolences!

[profiterole_reads]

I'm very impressed by your level of communication with your users. That's a huge change from LJ.

[denise]

I mean, I can either type it once in dw_maintenance or type it a thousand times individually!

[fyreharper]

Right, but the fact that you see those as the two options (vs ‘just not communicate’) says a lot!

[sara]

Dreamwidth: the photo hosting is pretty jank but you can get Dee on the phone.

[denise]

No you can't, I don't answer the phone. That would involve speaking out loud to a human :P

[sara]

...the phone is a metaphor.

I'm the last human who makes phone calls which is why people find me so extremely disarming. They're already off balance because the phone rang, and then I launch into my spiel.

Although R. was informed last week by the younger teen that "nobody texts anymore" so I'm not sure what the young people are doing nowadays, if nobody talks and nobody texts. Crimes, probably.

[ilyena_sylph]

Snapchat, mostly, my no-longer-teenager stepteen says.

[sara]

Locally Discord seems to be the blood of life but I'm sure it varies quite a bit.

[peaceflower]

Thank you for all the hard work. I didn't have anything to import when I moved over recently. Very happy to be here and never leaving. Happy to see more people moving over.

[graycardinal]

What boggles my mind is the concept of a journal that's generated >300K comments....

[pauses, counts on fingers]

Okay, Yuletide admin certainly, but I am clearly not traveling on DW's main highways through the infobahn....

[batrachian]

I am aware of at least a few anonymous-comment communities here on DW that generate five or six thousand comments per post.

Thank goodness for flat view.

[denise]

The one that we're babysitting along in the background right now is 7 million!

7 million comments

[elderwitty]

:blinks rapidly:

elderwitty.exe failed. Please try statement again.



Day-um!

[rifleman_lincoln]

Oh that's good news!

[zero_pixel_count]

I'm not even trying to import anything but in the spirit of IT solidarity, good work (both technical & comms).

[bikergeek]

I imported my LJ long ago, when LJ first moved their servers to Russia, but thank you for your ongoing efforts.

[lollycalifornia]

Thank you for all of your hard work, as always.

[acelightning73]

I imported all my LJ content when I first tried DW, and some of it is still here. And in this world's geopolitical situation, I'd rather not deal with anything that's based in Russia. (They censored some of my content back in the day. It was just a tiny bit too adult for Eastern European prudery.)

[denise]

That is something we absolutely won't be able to solve immediately, because it sounds like a problem with the data feed coming from LJ. Please open a support request in the Importer category and copy and paste that ^^ and add that Denise told you to say to leave the support request open until we get a chance to look at it!

[honeymink]

Thank you for your ongoing efforts. I was able to import my entries and comments successfully but I have a question – I had imported entries and comments before throughout the years, yet I found that the import process starts again every time from scratch. Would it be possible to say implement some measures where you can import things that haven't been imported already but leave things that are already imported instead of overriding them (I figure that's what's happening)? Wouldn't that keep some strain off the import queue?

[denise]

We do what we can to avoid re-downloading things, but are very much at the mercy of the data feed from LJ and what it allows us to request (and not request).

[honeymink]

I see. I just naïvely thought it could be managed in a way where one could say "I only want entries and comments from 2022." for instance. That would make it easier and the data volume not so high if you had diligently backed up everything before.

[denise]

Oh good Lord no, that would be far too easy. The API for journal downloading is the one Brad wrote in like 2000 and it's been touched like four times since then (and those were all mostly to fix compatibility problems with newer versions of Perl modules, heh). Getting data out of LJ is like emptying a lake with a bucket! It's just that everywhere else on the internet doesn't even give you the bucket, so it still is easier to interact with than like, Tumblr. (We tried to look into what Tumblr import would involve once and backed away very, very slowly without making eye contact.)

And did I mention it used to be badly documented, and is now completely undocumented because LJ deleted all the documentation? It is a miracle the importer can work at all and the only reason it does is that we knew exactly what we were working with from our time working on LJ, basically. I genuinely think anyone else trying to implement a similar feature would have broken down sobbing by the fourth time of having to completely recreate text encodings from the ground up because of a dumbfuck decision the Perl maintainers made in 1999 that caused Brad to make a "it made sense at the time even if in retrospect it looks highly dumbfuck" decision in 2000 and poison half the data in nearly every LJ created before, like, 2005. Just as an example.

The importer basically only exists at all because we are very very fucking stubborn and refuse to let the goddamn thing defeat us. We have made it work this far, goddammit, we are not losing now.

Strong Work

[degringolade]

As always, thank you.

But Lord!!! >300,000 comments????

I cringe at the thought. Never want to be that popular.

[thatwasjustadream]

thanks for everything you're doing! I was able to 100% port over a comm the other week including comments. Now I'm going in 1x per week to update entries/comments, but comments keep failing. We are only talking about a handful of comments but can't get them over. Same issue(s) as those trying to port over large numbers of comments?

Will hold off a few days after this one fails out and try later in the week.

[denise]

oh god I know from just reading your description this is going to wind up being either a three second fix or some horrible thing that takes a week and a half, lol. I've put the account on the list of things to look into!

[thatwasjustadream]

So sorry! I hope not the later. :). Or just kick it to the back of the queue if it looks to be the case!

[smutmods]

Thank you folks. <3

Just gonna leave this here...

[but_can_i_be_trusted]

💗

[matsushima]

Thank you for all your hard work on this!

[yourlibrarian]

Very good news!

As there are so many people doing imports right now, I hoped to signal boost this issue: polls done on LJ do not copy over. People will need to copy and paste those manually.

I realize that in most cases people won't mind losing them as the polls were done spontaneously for some personal or temporary reason. But some others were not and contain great snapshots of what a fandom or group of fans were thinking and doing at the time. So I really want to encourage people to not lose those results pages for important surveys they did.

[denise]

Oh good call! It's been long enough since I imported that I lost track of what we can't bring over. (Journal embeds, too, like links to YouTube videos: because reasons, they'll be replaced by a link to a totally random other thing. This is occasionally hilarious but always annoying, and I wish we could fix it but we can't.)