dw_maintenance | The importer is still running, we swear! It's just very busy.

The importer is still running, we swear! It's just very busy.

People who are worried because the import of their LiveJournal account has been running for a long time with no real signs of progress: please don't be concerned! The import queue is currently a little long. (In the same way that Mt Everest is a little tall and the Mariana Trench is a little deep.) We're limited in how many import jobs we can run simultaneously and how quickly we can start the next import after one finishes: LiveJournal, like all sites, has restrictions on how frequently we can programmatically request data from their site, so the import queue can get very backed up at times like this when more imports are being started than are finishing. If you look at the import queue and the numbers don't seem to be changing much, or are only going up, it doesn't mean that no imports are finishing: it means a lot of additional people have scheduled an import since the last time you reloaded.

As long as you haven't gotten a failure message in your on-site inbox, your import is still running. (Even if you have gotten a failure message, your import may still be running: if the site thinks that the failure is something that might correct itself, like being unable to connect to the remote site, it will retry for a few times before giving up.) If you have gotten a failure message, the error message in your inbox should tell you what went wrong.

The three common problems right now: 1) you mistyped your username and/or password; 2) you need to agree to LiveJournal's new ToS before they'll permit you to access the data in your account; 3) an entry or entries in your LiveJournal account have a text encoding mismatch and you need to follow the link in the error message to fix it on LJ.

If you haven't gotten a failure message, your import is still waiting in the queue, and will run when it makes its way up to the top of the queue.

People keep asking us how long the queue is (by which they mean, how much time will it take for a job just started to successfully finish: length of time, not number of jobs waiting). I would love to be able to give you a definite answer! It's really, really hard for us to predict how long it will take for a job to get up to the top of the queue, though: how long an import takes to complete depends on a lot of things, including how many posts/comments are in the journal. To give you a ballpark figure that might be off by up to 100% on either side: If I personally started a brand new import right now (in my timezone, the early morning of Monday 10 Apr), I would be pleasantly surprised if it finished before Tuesday morning (24 hours or so), would expect it to finish sometime on Tuesday night or maybe even stretch all the way to Wednesday night (36-60 hours), and wouldn't start to wonder if I should poke

mark or

alierak to doublecheck that something hadn't gotten stuck in such a way that our monitoring didn't alert us about it until Thursday afternoon or evening (100+ hours).

All of those time estimates, by the way, assume a relatively uncomplicated job that succeeds on the first try. When the site tries again after a failure, it includes a delay that increases after each failure in case the failure was due to transient network issues. So, I know some of you started an import at the end of last week and it's still running: some of you are trying to import very large journals, and some of you ran into errors along the way and are in a retry wait loop. Again: if you haven't gotten the final error message in your inbox (and it will tell you it's the final error), it's still chugging along.

You do not have to leave the importer page open or stay logged into Dreamwidth until your import finishes. (You do have to avoid changing your LJ password until the job is done, or it will fail.) You can close the window/tab and go off and explore Dreamwidth; the movers will be along in a little while with your stuff.

The tl;dr version of my usual longwinded babble: IMPORTER VERY BUSY. MANY PEOPLE MOVING IN. LIKE ON DORM OR APARTMENT MOVE-IN DAY, FREIGHT ELEVATORS VERY SLOW. BUILDING OWNERS RUNNING FREIGHT ELEVATORS AS FAST AS POSSIBLE AND APOLOGIZE FOR THE WAIT.

A housewarming glass of champagne/sparkling cider/fancy handmade soda for all! Welcome to the neighborhood.

EDIT, 10 Apr 2017 7:15PM EDT: the importer is not the only thing that is very busy today! I'm trying to get to all the comments here, but keep getting dragged off to handle other stuff (and will be knocking off for the night soon). If you have a technical support problem, it will probably be faster to open a support request, where there's less likelihood that it will get overlooked in the sea of comments.

Flat | Top-Level Comments Only | Expand All

Thanks for reminding us all what it's like to be on a site that actually cares about its users!

I just don't want people to think that we can't handle the load of a bunch of new users! We totally can scale up usage with zero problems, it's just that we have to throttle the importer down to "within the limits that LJ places on people who want to programmatically access the site's API", and that means we can't just start up extra workers to clear out the queue faster. :(

Oh, I completely get that. I'm amazed at what you've managed to do considering how many people are currently jumping off the wreck that is LJ... Thank you so much for all your work.

Fortunately, we have a lot of experience in this sort of thing!

mark and I realized a few weeks ago that if you count the time we were working on LJ, we've both been doing this for like over fifteen years each. Like, not "working in the tech industry" sort of "doing this for fifteen years"; I mean, I started working on LJ at the tail end of 2001 as a volunteer (and got hired at the end of 2002), and, barring like one eight-month period where I was hiding under the bed drinking heavily after I quit LJ in 2007 and before we got the idea for DW and start working on it in 2008, I've been working on this thing ever since. (Mark was 2002-2006 and 2008-now; he thought he'd escaped until I messaged him with "So, I have this ludicrous idea...".)

While this fact inevitably makes me feel dreadfully old, it also means that pretty much no matter what happens, we've been through it at least once before. :)

Holy wow. This makes me excited to be here. The loss of a permanent account hurts though.

Yeah, my lj permanent account paid itself off over ten years ago but it's still a bit sad. I remember putting the check addressed to LJ in the mail when I was a teen still.

Ugh, likewise. Mine was an early high school graduation present to myself. I will say that it helps me to remember that Dreamwidth is a sit I want to actively support with more money, but I do still feel the loss.

Heh, I don't mean to distract from the very real saudade we're all feeling, but I do have to say that holy fuck am I glad LJ didn't exist when I was a teen! Having my early 20s preserved forever is bad enough. ;) (I was just on the edge of the digital-childhood generation; my peers and I didn't get "proper" internet access until college for the most part and our teen years were spent on BBSes that were ephemeral enough that my teenage bad life choices are not immortlized forever. Thank God.)

Ahem.

All joking aside, though, I know exactly what people are feeling, or at least to an extent; I stuck it out with working for Six Apart (owners at the time I left) long past the point where I should have given up for my health and my sanity, because of how much I loved LJ, and even though by the time I quit I was swearing I would never work on anything even in the same neighborhood as social media ever again, it was eight months later when

mark and I started having the conversations that turned into DW. But no matter how amazing it is that people were willing to take a gamble on us in the beginning and have proven our wild-eyed idealism would actually work when put into practice, and no matter how glad I am that DW exists and can give people that escape hatch/lifeboat of saving their digital history even if they don't decide to shift entirely to DW from LJ/Facebook/Tumblr/etc, it's still incredibly sad to watch something we all poured so much of ourselves into start to crumble like this, isn't it?

I mean, DW isn't LJ, and will never be LJ (and we've never even tried to be LJ; we're us, not them, even if "them" used to be our younger selves once upon a time). So even if the vast majority of people who've ever used LJ did move their stuff over, which is unlikely, it's never going to be the same thing. (And that's not even getting into how many of my friends have died in the past 15 years and whose journals may get closed at any point from here on out, now, according to the new LJ ToS; I'm trying really hard not to think about that!) It's understandable that people are sad about it. Hell, I'm sad about it!

I was fifteen or so! IMAGINE THAT ONLINE. FOREVER. I privated the entries a long time ago but it feels like I have a horribly embarrassing ghost locked up inside the journal.

I like DW and I've been using it for a while. It keeps the format I enjoy with a lot of the things I didn't enjoy gone. Also a lot more ease of use and nice features like the double sticky.

And I feel you about the accounts.

Oh, Lord. Fifteen was not a good look on ANYONE. (I am occasionally very thankful that I was not born later than I was. Even my sister, who was born 5.5 years after me, only escaped having an embarrassing digital footprint by just not being much interested in online stuff back then; my parents got internet access at home just when she was going into high school. If our birth order had been switched, I'd've been the one being fifteen all over USENET or something.)(As it was, I did get USENET access along with internet access when I first hit college, and oh boy am I thankful that the worst of that isn't still online...)

(no subject)

crantz - 2017-04-12 06:31 (UTC) - Expand

(no subject)

denise - 2017-04-12 08:41 (UTC) - Expand

I hear ya. I mean, even though I'm glad DW exists (and would be even if I weren't a co-owner) for people to move to, I'm still tremendously sad to watch LJ's long slow meltdown. I put a lot of my life into it...

Thank you so much to you and mark both for starting DW and never giving up on it! LJ was my first home on the internet, but DW really is my home now. And the venture capital startup model can go -- whistle.

Oh, Lord, if you get me started on all of the moral, cultural, and socioeconomic failings of the prevailing business model of like the entire tech industry we'll be here all day. Suffice it to say that I like our way better, too.

LJ was my first home on the Internet too (aside from a MUX/IRC!). I spent a lot of years pouring everything I had into LJ -- blood, sweat, and tears.

But here we are and here we'll stay. I love our new little corner of the Internet and I'm glad you're here. :)

Speaking for myself, I thank you for coming out of hiding. Especially for this. No reply needed.

Edited 2017-04-10 23:53 (UTC)

Aww, thanks! Yeah, I'm glad I did too, even though I really needed those eight months of mental health break at the time...

Thank you SO MUCH for starting Dreamwidth. I don't know what I'd do if y'all didn't exist.
(PS: you do not have to respond to this comment. You are swamped.)

We totally can scale up usage with zero problems

Just don't sell DW to Russian government, like LJ owners did and don't treat Tor/i2p connections badly, like Google does...

We have no interest in selling to anybody, don't worry! We're stubborn. :)

As for Tor: We don't automatically block Tor exit nodes. We do use Cloudflare for DDoS protection. (We don't get DDoSed *regularly*, but we do get it occasionally and when we do, Cloudflare is what keeps us up.) Cloudflare also doesn't automatically block all Tor exit nodes, but they do block some exit nodes that have particularly bad histories, and sometimes we have to use Cloudflare's tools to block more of them when someone flooding or DDoSing us is using Tor.

In other words: we are pro-Tor in theory, since it's such a useful privacy tool! In practice, we sometimes need to restrict accessing the site via Tor, because much of the malicious traffic we get comes through Tor: not all Tor traffic is malicious, of course, but 99% of our malicious traffic comes via Tor and sometimes we have to implement restrictions in order to stop it.

In practice, we sometimes need to restrict accessing the site via Tor, because much of the malicious traffic we get comes through Tor: not all Tor traffic is malicious, of course, but 99% of our malicious traffic comes via Tor and sometimes we have to implement restrictions in order to stop it.

May be it's better to create dreamwidthXXXXX.onion and handle Tor traffic as a native Tor service?

Huh! That's an interesting idea. I'm not sure we have the resources to do it (it would definitely require a lot of research first, since none of us are Tor experts) but I'll put it on the list of Things To Think About In The Future.

Yes, consider this: if you serve native Tor and block Tor exit nodes (these are anyway could be malicious and could do much more harm besides dumb DDoS) then your main system is not going to be affected anymore by Tor attacks. On other hand, if someone somehow DDoSes your Tor subsystem, only that service is going to be down.

From Tor FAQ:

What about distributed denial of service attacks?

Distributed denial of service (DDoS) attacks typically rely on having a group of thousands of computers all sending floods of traffic to a victim. Since the goal is to overpower the bandwidth of the victim, they typically send UDP packets since those don't require handshakes or coordination.

But because Tor only transports correctly formed TCP streams, not all IP packets, you cannot send UDP packets over Tor. (You can't do specialized forms of this attack like SYN flooding either.) So ordinary DDoS attacks are not possible over Tor. Tor also doesn't allow bandwidth amplification attacks against external sites: you need to send in a byte for every byte that the Tor network will send to your destination. So in general, attackers who control enough bandwidth to launch an effective DDoS attack can do it just fine without Tor.

Thanks for the idea!

Thanks for reminding us all what it's like to be on a site that actually cares about its users!

Right? That's how I feel as well. It's one of the main reasons I moved here. Maybe THE main reason, back a few years ago!

After being a paying user for YEARS at That Other Site That Shall Not Be Named, I had some issue or another and was treated like crap and told if I didn't like it, I could leave.

So I took them at their word and eventually did just that.

Everyone at Dreamwidth has always been so communicative and supportive from the first, I love it. It's super transparent here.

Flat | Top-Level Comments Only | Expand All