dw_maintenance | The importer is still running, we swear! It's just very busy.

The importer is still running, we swear! It's just very busy.

People who are worried because the import of their LiveJournal account has been running for a long time with no real signs of progress: please don't be concerned! The import queue is currently a little long. (In the same way that Mt Everest is a little tall and the Mariana Trench is a little deep.) We're limited in how many import jobs we can run simultaneously and how quickly we can start the next import after one finishes: LiveJournal, like all sites, has restrictions on how frequently we can programmatically request data from their site, so the import queue can get very backed up at times like this when more imports are being started than are finishing. If you look at the import queue and the numbers don't seem to be changing much, or are only going up, it doesn't mean that no imports are finishing: it means a lot of additional people have scheduled an import since the last time you reloaded.

As long as you haven't gotten a failure message in your on-site inbox, your import is still running. (Even if you have gotten a failure message, your import may still be running: if the site thinks that the failure is something that might correct itself, like being unable to connect to the remote site, it will retry for a few times before giving up.) If you have gotten a failure message, the error message in your inbox should tell you what went wrong.

The three common problems right now: 1) you mistyped your username and/or password; 2) you need to agree to LiveJournal's new ToS before they'll permit you to access the data in your account; 3) an entry or entries in your LiveJournal account have a text encoding mismatch and you need to follow the link in the error message to fix it on LJ.

If you haven't gotten a failure message, your import is still waiting in the queue, and will run when it makes its way up to the top of the queue.

People keep asking us how long the queue is (by which they mean, how much time will it take for a job just started to successfully finish: length of time, not number of jobs waiting). I would love to be able to give you a definite answer! It's really, really hard for us to predict how long it will take for a job to get up to the top of the queue, though: how long an import takes to complete depends on a lot of things, including how many posts/comments are in the journal. To give you a ballpark figure that might be off by up to 100% on either side: If I personally started a brand new import right now (in my timezone, the early morning of Monday 10 Apr), I would be pleasantly surprised if it finished before Tuesday morning (24 hours or so), would expect it to finish sometime on Tuesday night or maybe even stretch all the way to Wednesday night (36-60 hours), and wouldn't start to wonder if I should poke

mark or

alierak to doublecheck that something hadn't gotten stuck in such a way that our monitoring didn't alert us about it until Thursday afternoon or evening (100+ hours).

All of those time estimates, by the way, assume a relatively uncomplicated job that succeeds on the first try. When the site tries again after a failure, it includes a delay that increases after each failure in case the failure was due to transient network issues. So, I know some of you started an import at the end of last week and it's still running: some of you are trying to import very large journals, and some of you ran into errors along the way and are in a retry wait loop. Again: if you haven't gotten the final error message in your inbox (and it will tell you it's the final error), it's still chugging along.

You do not have to leave the importer page open or stay logged into Dreamwidth until your import finishes. (You do have to avoid changing your LJ password until the job is done, or it will fail.) You can close the window/tab and go off and explore Dreamwidth; the movers will be along in a little while with your stuff.

The tl;dr version of my usual longwinded babble: IMPORTER VERY BUSY. MANY PEOPLE MOVING IN. LIKE ON DORM OR APARTMENT MOVE-IN DAY, FREIGHT ELEVATORS VERY SLOW. BUILDING OWNERS RUNNING FREIGHT ELEVATORS AS FAST AS POSSIBLE AND APOLOGIZE FOR THE WAIT.

A housewarming glass of champagne/sparkling cider/fancy handmade soda for all! Welcome to the neighborhood.

EDIT, 10 Apr 2017 7:15PM EDT: the importer is not the only thing that is very busy today! I'm trying to get to all the comments here, but keep getting dragged off to handle other stuff (and will be knocking off for the night soon). If you have a technical support problem, it will probably be faster to open a support request, where there's less likelihood that it will get overlooked in the sea of comments.

Flat | Top-Level Comments Only

Just as I was checking on my import progress, this is posted! 😊

I completely understand. The queue depth for comments, which I'm waiting on, is currently 752.

💗

And it's 768 now! THERE IS NO SLOW TIME ANYMORE, heh.

(Comments are awfully rough to begin with, since most people have a lot of 'em, it takes a while to process them properly on our end much less download them from LJ, and the processing can't even start until entries are imported. So they're always going to take longer than entries.)

Oh yeah, there is a lot, because I'm importing from a community!

My personal journal import will take a lot less time.

Ohhh, yeah, communities usually take a lot longer to process, too!

Something I only realized after I did it was that a few of my old posts are absolutely choked with spam, as in, hundreds or possibly thousands of garbage porn-spam comments on one post (I had anonymous comments enabled but screened). One thing Dreamwidth doesn't have that LJ eventually did get around to adding is a "nuke all screened/suspicious comments" button, so it probably would have been better for me to notice that before the import.

At any event, I bit the bullet and restricted commenting to registered accounts on Dreamwidth.

Edited 2017-04-10 15:43 (UTC)

Uggggghhhhhh. Yeah, we haven't had to build in stronger automated spam cleanup tools than the ones LJ had in '08 when we forked, because we haven't been a tempting enough target for spammers that the spam volume has increased past the point where our antispam team could handle it themselves through those tools. :(

If you don't want to hand-delete all of the spam, you can delete all your imported entries, go back to LJ, use their cleanup tools, and then re-import. (I think it will also work if you delete just the entries with the worst spam, go over to LJ, use the cleanup tools, and reimport; I'm pretty sure the importer will recognize that the post is missing and re-create it, and even if I'm wrong, the failure state would be "fails to re-import the post(s) that was deleted, requiring you to wipe everything and re-import if you want to make sure you got everything", not "duplicates everything". I'd read the code to be 100% sure instead of just pretty sure, but it would take me too long!)

DW STAFF VERY COMMUNICATIVE and we LOVE YOU!!!

(I'm not currently importing anything, I deleted my LJ about five years or so ago, but I just wanted to say how much we appreciate you for keeping us in the loop. To those newly arrived from LJ: See! DW staff TALK TO US!! Be excite!)

DW STAFF OCCASIONALLY TALK IN PUNCH-DRUNK HULK SMASH ALLCAPS BECAUSE OH MY GOD THIS WEEK

(I would also warn "DW staff have pottymouths and often forget to self-censor into business-appropriate communication tone", but that would unfairly indict

mark and

karzilla, who are perfectly capable of not swearing like a New Jerseyean without even having to think about it too hard. So, really, it's more "

denise was born and raised in New Jersey and doesn't always realize she's using profanity as punctuation again".)

Having figured out I can actually afford six months' paid time, I will just point to my icon... :D

Thank you for helping to subsidize the site for people who can't afford paid time!

Bearing in mind it's been nearly a year since I've been able to afford paid time, and it's only six months' worth. DW is something I do feel I want to support, too.

Also, I missed having my full range of icons. ;)

"Profanity as communication" is best way to communicate. Axel has caught Hulk language from reading DW Staff comments. Good job, Denise!

I think of the pottymouth as me doing people a favor, really. If someone's the type to get offended at my tendency to treat 'fuck' as an all-purpose word, they're probably also the type who will not appreciate any of our general tone/register of communication! So my complete failure to ever behave like a proper serious-business grownup adult for more than five minutes is really a benefit.

(That's my story and I'm sticking to it, anyway.)

One of those moving in and I appreciate the update! I'm so glad dw offers the free import in the first place, I'm in no hurry! I don't feel like unpacking my moving boxes until I return from my Easter vacation anyway. ;-)

*leaves a sixpack of beers for the movers for when they arrive with the moving boxes from lj*

Edited 2017-04-10 10:40 (UTC)

Save the beer for the giant block party we'll throw once everybody's finished moving into the neighborhood! I'll start a pot of chili simmering now.

I am speaking as an LJ refugee, thank you.

/me sneaks a beer :o

when things calm down, Mark, I will buy you a whole keg if you want. :)

Thanks for reminding us all what it's like to be on a site that actually cares about its users!

I just don't want people to think that we can't handle the load of a bunch of new users! We totally can scale up usage with zero problems, it's just that we have to throttle the importer down to "within the limits that LJ places on people who want to programmatically access the site's API", and that means we can't just start up extra workers to clear out the queue faster. :(

Oh, I completely get that. I'm amazed at what you've managed to do considering how many people are currently jumping off the wreck that is LJ... Thank you so much for all your work.

Fortunately, we have a lot of experience in this sort of thing!

mark and I realized a few weeks ago that if you count the time we were working on LJ, we've both been doing this for like over fifteen years each. Like, not "working in the tech industry" sort of "doing this for fifteen years"; I mean, I started working on LJ at the tail end of 2001 as a volunteer (and got hired at the end of 2002), and, barring like one eight-month period where I was hiding under the bed drinking heavily after I quit LJ in 2007 and before we got the idea for DW and start working on it in 2008, I've been working on this thing ever since. (Mark was 2002-2006 and 2008-now; he thought he'd escaped until I messaged him with "So, I have this ludicrous idea...".)

While this fact inevitably makes me feel dreadfully old, it also means that pretty much no matter what happens, we've been through it at least once before. :)

Holy wow. This makes me excited to be here. The loss of a permanent account hurts though.

Yeah, my lj permanent account paid itself off over ten years ago but it's still a bit sad. I remember putting the check addressed to LJ in the mail when I was a teen still.

Ugh, likewise. Mine was an early high school graduation present to myself. I will say that it helps me to remember that Dreamwidth is a sit I want to actively support with more money, but I do still feel the loss.

Heh, I don't mean to distract from the very real saudade we're all feeling, but I do have to say that holy fuck am I glad LJ didn't exist when I was a teen! Having my early 20s preserved forever is bad enough. ;) (I was just on the edge of the digital-childhood generation; my peers and I didn't get "proper" internet access until college for the most part and our teen years were spent on BBSes that were ephemeral enough that my teenage bad life choices are not immortlized forever. Thank God.)

Ahem.

All joking aside, though, I know exactly what people are feeling, or at least to an extent; I stuck it out with working for Six Apart (owners at the time I left) long past the point where I should have given up for my health and my sanity, because of how much I loved LJ, and even though by the time I quit I was swearing I would never work on anything even in the same neighborhood as social media ever again, it was eight months later when

mark and I started having the conversations that turned into DW. But no matter how amazing it is that people were willing to take a gamble on us in the beginning and have proven our wild-eyed idealism would actually work when put into practice, and no matter how glad I am that DW exists and can give people that escape hatch/lifeboat of saving their digital history even if they don't decide to shift entirely to DW from LJ/Facebook/Tumblr/etc, it's still incredibly sad to watch something we all poured so much of ourselves into start to crumble like this, isn't it?

I mean, DW isn't LJ, and will never be LJ (and we've never even tried to be LJ; we're us, not them, even if "them" used to be our younger selves once upon a time). So even if the vast majority of people who've ever used LJ did move their stuff over, which is unlikely, it's never going to be the same thing. (And that's not even getting into how many of my friends have died in the past 15 years and whose journals may get closed at any point from here on out, now, according to the new LJ ToS; I'm trying really hard not to think about that!) It's understandable that people are sad about it. Hell, I'm sad about it!

I was fifteen or so! IMAGINE THAT ONLINE. FOREVER. I privated the entries a long time ago but it feels like I have a horribly embarrassing ghost locked up inside the journal.

I like DW and I've been using it for a while. It keeps the format I enjoy with a lot of the things I didn't enjoy gone. Also a lot more ease of use and nice features like the double sticky.

And I feel you about the accounts.

Oh, Lord. Fifteen was not a good look on ANYONE. (I am occasionally very thankful that I was not born later than I was. Even my sister, who was born 5.5 years after me, only escaped having an embarrassing digital footprint by just not being much interested in online stuff back then; my parents got internet access at home just when she was going into high school. If our birth order had been switched, I'd've been the one being fifteen all over USENET or something.)(As it was, I did get USENET access along with internet access when I first hit college, and oh boy am I thankful that the worst of that isn't still online...)

'Oh no... oh no' as you read old entries.

YES. Along with putting your hands over your face and reading gingerly along by peeking through your fingers...

I hear ya. I mean, even though I'm glad DW exists (and would be even if I weren't a co-owner) for people to move to, I'm still tremendously sad to watch LJ's long slow meltdown. I put a lot of my life into it...

Thank you so much to you and mark both for starting DW and never giving up on it! LJ was my first home on the internet, but DW really is my home now. And the venture capital startup model can go -- whistle.

Oh, Lord, if you get me started on all of the moral, cultural, and socioeconomic failings of the prevailing business model of like the entire tech industry we'll be here all day. Suffice it to say that I like our way better, too.

LJ was my first home on the Internet too (aside from a MUX/IRC!). I spent a lot of years pouring everything I had into LJ -- blood, sweat, and tears.

But here we are and here we'll stay. I love our new little corner of the Internet and I'm glad you're here. :)

Speaking for myself, I thank you for coming out of hiding. Especially for this. No reply needed.

Edited 2017-04-10 23:53 (UTC)

Aww, thanks! Yeah, I'm glad I did too, even though I really needed those eight months of mental health break at the time...

Thank you SO MUCH for starting Dreamwidth. I don't know what I'd do if y'all didn't exist.
(PS: you do not have to respond to this comment. You are swamped.)

We totally can scale up usage with zero problems

Just don't sell DW to Russian government, like LJ owners did and don't treat Tor/i2p connections badly, like Google does...

We have no interest in selling to anybody, don't worry! We're stubborn. :)

As for Tor: We don't automatically block Tor exit nodes. We do use Cloudflare for DDoS protection. (We don't get DDoSed *regularly*, but we do get it occasionally and when we do, Cloudflare is what keeps us up.) Cloudflare also doesn't automatically block all Tor exit nodes, but they do block some exit nodes that have particularly bad histories, and sometimes we have to use Cloudflare's tools to block more of them when someone flooding or DDoSing us is using Tor.

In other words: we are pro-Tor in theory, since it's such a useful privacy tool! In practice, we sometimes need to restrict accessing the site via Tor, because much of the malicious traffic we get comes through Tor: not all Tor traffic is malicious, of course, but 99% of our malicious traffic comes via Tor and sometimes we have to implement restrictions in order to stop it.

In practice, we sometimes need to restrict accessing the site via Tor, because much of the malicious traffic we get comes through Tor: not all Tor traffic is malicious, of course, but 99% of our malicious traffic comes via Tor and sometimes we have to implement restrictions in order to stop it.

May be it's better to create dreamwidthXXXXX.onion and handle Tor traffic as a native Tor service?

Huh! That's an interesting idea. I'm not sure we have the resources to do it (it would definitely require a lot of research first, since none of us are Tor experts) but I'll put it on the list of Things To Think About In The Future.

Yes, consider this: if you serve native Tor and block Tor exit nodes (these are anyway could be malicious and could do much more harm besides dumb DDoS) then your main system is not going to be affected anymore by Tor attacks. On other hand, if someone somehow DDoSes your Tor subsystem, only that service is going to be down.

From Tor FAQ:

What about distributed denial of service attacks?

Distributed denial of service (DDoS) attacks typically rely on having a group of thousands of computers all sending floods of traffic to a victim. Since the goal is to overpower the bandwidth of the victim, they typically send UDP packets since those don't require handshakes or coordination.

But because Tor only transports correctly formed TCP streams, not all IP packets, you cannot send UDP packets over Tor. (You can't do specialized forms of this attack like SYN flooding either.) So ordinary DDoS attacks are not possible over Tor. Tor also doesn't allow bandwidth amplification attacks against external sites: you need to send in a byte for every byte that the Tor network will send to your destination. So in general, attackers who control enough bandwidth to launch an effective DDoS attack can do it just fine without Tor.

Thanks for the idea!

Thanks for reminding us all what it's like to be on a site that actually cares about its users!

Right? That's how I feel as well. It's one of the main reasons I moved here. Maybe THE main reason, back a few years ago!

After being a paying user for YEARS at That Other Site That Shall Not Be Named, I had some issue or another and was treated like crap and told if I didn't like it, I could leave.

So I took them at their word and eventually did just that.

Everyone at Dreamwidth has always been so communicative and supportive from the first, I love it. It's super transparent here.

I've been mentioning to friends transferring that the word needed it just 'persevere'.

Those of us belonging to certain minorities have been left with no choice (although, in truth, I've been based here since 2012).

Yup, fear not, once you schedule an import, it WILL get handled! ...eventually. :)

I have had a free DW account for years, but after giving up my LJ, I definitely intend to get a paid account as well. Why? Because open communication deserves to be supported :) *tips hat to DW staffers*

*salutes you back with the 2L bottle of cherry coke zero I am subsisting on*

And thank you for supporting us! We will never be as slick or as fast-developing or as fancy as some sites out there that subsist off venture capital, but for a couple of people who can best be described as "fluffy idealists" sometimes, it's been really wonderful over the past 8 years (oh god eight years) to see that our idealism wasn't unfounded after all and it really is possible to do this without conforming to the standard startup model.

I hope that by now you're subsisting on more than calorie free soda! Thank you for all the hard work you're putting in with this new influx.

I am literally sitting down to dinner with my lovely wife as I type this!

I completely agree, and am planning on moving to a paid account as soon as I'm able.

LiveJournal's failure is your success — though I'm sure you wish success wouldn't pile on quite so quickly. :)

There are sooooo many bad jokes I can make here, but pretty much all of them fall into the category of Things Denise Should Not Say In Public Because She's Usually Not As Funny As She Thinks She Is And The Failure Mode Of Clever Is Asshole...

Seriously, though, we're thriled to have all y'all who are moving in this week, whether as a "I'm just gonna use my DW account to back up the contents of my LJ account in the event of future disaster" all the way along the spectrum to to I AM THELMA AND LOUISE'ING MY LIVEJOURNAL STRAIGHT OFF THE CLIFF, SEE YOU ON DW ONCE THE DEBRIS CLEARS. I hope that people who are used to the LJ way of doing things can adapt to the DW style over time! (Said style basically being "slower development, sustainable business practice, and run by Real People who occasionally have trouble remembering where they left their pants".)

Nothing has described how I left LJ like "I AM THELMA AND LOUISE'ING MY LIVEJOURNAL STRAIGHT OFF THE CLIFF, SEE YOU ON DW ONCE THE DEBRIS CLEARS."

Same here! *chortles*

I'm glad that my slightly-punchy-and-a-bit-sleep-deprived babbling at least has given you a new rallying cry!

The metaphor works so well for far too many of us, doesn't it?

Word. Exactly how mine went, too! Well, pretty much...

DW: Like LJ used to be in the good ol' days, minus lot of fandom who jumped to Tumblr, dang it all.

So very true. (I just find Tumblr near-inaccessible, between the constant gifs and the tiny tiny font a lot of people use, and the way they don't have comments on posts but instead those added remarks that go to one-letter-per-line very quickly.)

Yeah. I really don't care for the tumblr model either - there's so little discussion, just a stream of "so-and-so liked this/thought it was worth passing along" with no elaboration.

Yeah, and it's really impossible for me to find pretty much anything, via search or tags, that's more than a couple of days old. Even with Google! And people change their names so often, that's a factor too. LOL I sound so old

Did you look in the refrigerator for your pants?

omg long time no see, you!

For those in the queue: I usually import my comments every 6 months and it takes a few minutes. This week it took 12+ hours. But it got there eventually, and so will yours.

Data points are useful, thank you!

Agree with this, it's usually about 5 minutes or so whereas last week it was overnight.

Hmm. Interesting - so I can import just comments. Does that work even if the people commenting don't have DW accounts? *should just wander off to check the FAQ*

Thanks for the information!

You can! You do have to import your entries along with the comments, though, or else there's nothing to stick them on. :)

Imported comments are displayed as coming from "so-and-so.livejournal.com" and, thanks to the magic of OpenID, as long as you still have access to the LJ account you were logged into when you originally made the comment on LJ, you can log into DW using that OpenID account and still have the same level of control (edit, delete, etc) as you had for the LJ version.

Or, you can claim your OpenID with your DW account, and then all the comments you've left in other people's journals with that LJ account will update to display as your DW account instead. For instance, look at the comments to this post from 2008, which was originally made on LJ, and was the first time that we talked about DW in public. I imported the entry, and all its comments, from LJ; the first comment thread, now appearing to be from me and zorkian (which is mark's personal account like synecdochic is my personal account), originally displayed like how many of the others still do. (As xb95.livejournal.com and synecdochic.livejournal.com, respectively.) Then I claimed my synecdochic.livejournal.com OpenID URL with synecdochic, and he claimed his xb95.livejournal.com OpenID URL with zorkian, and now anywhere on the site that somebody has imported a comment that I made while logged into LJ as synecdochic, it shows as synecdochic instead!

(It's even smart enough that if you have the same icon keywords on DW as you did on LJ, after you claim your OpenID, all the comments will even display the proper icon.)

Hmmm, so. I ported my journal over when DW was born (I'm an early user!) and I've been cross-posting since then using Semagic. However, I've a load of comments on LJ (when it was more active) than I do here, because see: more active.

If I imported those comments, they'd come with the posts to which they're attached, which would double my posts, wouldn't it?

I'm not going to do anything with this right now, as the import system is understandably overwhelmed, but it's something to think about. It would be nice to keep those comments, if at all possible.

When you import, cross posts originally made on DW don't duplicate. I had comments on both sites and would import my comments every 3 to 6 months in order to have then all in the same place.

I wonder if that works for all crossposts - I use a third party program called Semagic.

I'll do some testing, when things are less chaotic.

Thank you!

Yeah, sorry, I had your comment open in a tab to answer and then it got lost among the EIGHT BILLION OTHER TABS I've got right now, heh.

If you were crossposting using Semagic, instead of using the Dreamwidth-native crosspost utility, unfortunately there's no way to import your LJ without duplicates resulting: the only way for the importer to know that a particular entry on both sites is the same actual entry is if the entry was generated through the crossposter. (We don't do things like comparing the actual entry content to see if two entries are the same, because text parsing on that level is annoyingly difficult and slow, and the smallest of differences between the two versions could result in entries not being merged when they should be -- or, worse, being merged when they shouldn't be.) The same thing applies if you were manually copying and pasting your entries into both sites' update pages.

My best suggestion is to create a separate account (something like stasia_from_lj or something) and import the current state of your LJ into it -- it might be annoying to have to look in two places for something you remember, but I'd personally find it less annoying than having two copies of all the entries from 2009-2017.

You're a Tab Collector as well? *high five* I've found this great chrome extension which makes them "sleep" while I'm not looking at them. It lessens their hit on memory and disk space. It's called The Great Suspender.

Anyway, yeah, I figured that was the issue. Having a second journal just to save those comments might be the best option. I'll wait, though, until the rush dies down.

That is amazing and I need to find out if there's a version for FireFox. I've got, like, twenty plus tabs from Tumblr open of "things I'm going to reblog as soon as I have the spoons to tag decently, I swear", and probably at least a third of those also require spoons for describing images and/or translating capslock.

omfg thank you so much for the rec! I force myself to viciously prune them at the end of the day, because omg, but it would be nice to not to have to...

Um, this may be why I have – oh god – over 78,000 bookmarks in my Pinboard. >_>

Sad high-five of fellow virtual packratness!

Just came back to say I installed this extension last night and am already noticing a difference. Thank you again!

Thanks for the update. I'm one of the people in the queue - and have a second journal to bring over when this one's done! - and I can only imagine the laod this is putting on you guys.

It's actually not terrible! I mean, theoretically we could clear out the queue a lot faster by increasing the number of worker processes running at once -- we have the CPU and RAM to spare for it -- but LJ throttles how many connections can be active at one time, and we try really hard to stay within the boundaries of their policies to avoid problems...

Welcome to DW, by the way!

So, what, has LiveJournal taken their remaining users’ data hostage? Is that even legal? (J/K, pretty sure it’s covered by the old ToS.)

Many APIs throttle how many connections can be made at one time, it's to protect servers and keep the server load somewhat consistent to avoid problems.

In non-tech terms, basically there are only so many exit doors to the LJ building, and we're bunched up waiting for our turn to get to a door.

Throttling I can understand, that’s just self-preservation. I was reacting to this:

The three common problems right now: […] 2) you need to agree to LiveJournal's new ToS before they'll permit you to access the data in your account; […]

Requiring to agree to the new ToS before releasing the data for migration — if that is true — when said ToS *is* the actual reason people are migrating — is actively evil.

To be scrupulously fair because that's how I try to roll, it doesn't necessarily have to be actively evil, just the law of unintended consequence. The way LJ (and DW, since we're a code fork of LJ) is programmed, the best way to keep someone from posting or commenting while logged in until they agree to a new version of the ToS just so happens to also fire when you're using the XML-RPC protocol to download journal contents, and DW's importer uses the XML-RPC protocol. That "you can't do anything until you agree to the new ToS version" was actually added ages ago. (I want to say ...2005? 2006? Something like that. While I was still working for LJ and it was still owned by Six Apart, at least.) So, it's possible the current team went to go build a ToS check, realized there was support for doing one already in the code, and enabled it, without specifically realizing/actively deciding that it would also block people from accessing their data through the XML-RPC protocol.

Mind you, I don't know for sure one way or the other! I just like to point out when the fail is possibly accidental instead of having to automatically be malicious. :)

Okay. Hanlon’s Razor strikes again.

I mean, I've been working with (some variety of) this codebase for like 16 years now. I KNOW HOW HARD IT IS TO FEEL CONFIDENT YOU ADEQUATELY UNDERSTAND ALL THE DETAILS OF WHAT WILL HAPPEN IF YOU CHANGE THAT ONE LINE. It's eighteen years' worth of legacy Perl! Sometimes I think it's a wonder it still works at all.

(This is why occasionally, we will do a code push, and five minutes later, we will make another code push consisting only of the emergency commit that has a commit message of just "lol oops" or similar...)

I actually must confess I have never had to proof my code against a malicious future owner. From a Lawful Neutral standpoint, not allowing any API operations to a non-ToS-consenting user is perfectly normal. It takes an actual safety-related designer to provide a fire escape.

There are pretty much only so many threat models you can design for at once...

"It's eighteen years' worth of legacy Perl!"

*eyetwitches*

I SEE YOU PROPERLY UNDERSTAND MY PAIN

I'd love to give LJ the benefit of the doubt on that one Denise, except apparently when the code update first went live (the one that made the TOS appear immediately after login), you could ignore the pop up and delete your account by going directly to the account page and change the account status there. That loophole got closed quite quickly though, so.....

Also if you look at their support board, their volunteers are now giving a canned response to people asking if LJ will delete their accounts since they can't do so without accepting the TOS. They are replying that LJ cannot delete an account, only the logged in user can. They must think people came down in the last shower. :/ I feel for the volunteers because they are only the messengers, but are getting some pretty angry replies.

Oh heck yeah, not arguing with that. I was NOT happy when I clicked the "not now" button on the TOS and LiveJournal tried to log me out. =(

So what are they going to do if you violate their TOS six ways from Sunday? Kick you off the service? I mean, you've already left by that point.

I'm a U.S. citizen who grew up during the Cold War, and Putin is an ex-KGB goon turned dictator. I wouldn't actually feel bad about breaking Russian laws regarding blogging or political speech or "pornography". Anyone got some NSFW Putin/Trump slash art we can cross-post to LJ as a "going-away present" once the transfers are done?

So, what, has LiveJournal taken their remaining users’ data hostage? Is that even legal? (J/K, pretty sure it’s covered by the old ToS.)

I know you're meaning in terms of allowing the importer, but actually LJ has taken hostage of people's data in the sense that you cannot edit or delete entries, or even delete your LJ account UNLESS you accept the new TOS. At the very least it would have been decent of them to have an option to allow people to delete their accounts without accepting the agreement if use of the site was contingent on that.

I mean, there was a loophole where you could delete by going directly to the account status page, but that got closed pretty quick.

I also heard that it's possible to delete your account without agreeing to the new TOS if you turn Javascript off, but I haven't tried it. Mostly because I couldn't be bothered to look up how ^^a

But yes, those were both by accident rather than benevolence on LJ's part.

More and more, I like the idea of "agree to new TOS, delete all but one last post, post XXX photoshopped Trump/Putin slash."

I'll be honest - I write slash fics, and if not for the fact that I have no idea what they're going to do with user privacy if I continue to post slash, I wouldn't mind leaving all my posts on my LJ so that whoever opens it and decides that it has to be purged, likely had to read my fics and being embarrassed about it. /Shrug/ LOL
Although someone yes photoshop and do it.

Thanks for the update! I'm one of the freshly-imported-from-LJ people. My import didn't, in fact, take much time--guessing I was one of the lucky ones. I appreciate you keeping folks up to date.

I'll take my champagne with Chambord for a nice Kir Royal, thanks.

From a 2008-era LJ refugee now nicely settled in here, welcome! I'll mix you up that Kir Royale and myself a French 75, and we can get day drunk and read fanfiction!

MAKE MINE A NICE DRAGON AGE EPIC

And would the lady like a French 75 or a Kir Royale to go with their hack-and-slash slash? ;)

The lady, sadly, has been informed in no uncertain terms by her doctors that she is not allowed to drink alcohol anymore if she likes the idea of having a liver. Sigh.

But I love vicariously experiencing good booze through other people who aren't taking like four potentially hepatotoxic meds all at once! So pick your favorite and have one in my honor.

Hurrah! I like win-win situations. Cheers! :)

What's a French 75? In any case, though, I accept your invitation. :-)

French 75s are a WWI era champagne cocktail and they are amazing. Two parts gin, one part each lemon juice and syrup, shake and strain into a flute, top up with the bubbly. Legend has it the name came from a WWI flying ace's declaration that it 'hits you like a French 75' - that being the slang for the Canon de 75 modèle 1897, a particuarly rapid-firing piece of Gallic artillery.

Fair warning, I'm a giant nerd about a lot of things, including booze. ;) Cheers! *clink*

Ha, you think you have more nerd cred than me? We'll just see about that... :)

I don't fight about cred, you're welcome to it. :) I just say I'm a giant nerd because I love to learn about all kinds of things, and have been known to go on about them, at length, with very little prompting. If there are things you're enthusiastic about, I would love to hear about them too!

Glad it was nice and zippy for you! And welcome home.

Aw, thanks! :-)

Newly moved over from LJ, this place sure sounds fun!

Glad to have you! And we certainly do try to have fun, although sometimes I should have my laptop taken away from me until my ADHD meds stabilize. ;)

Thanks for the update!

It's been an interesting week!

It took the importer for my entries from LJ about 36 hours to complete import, just FYI, guys. Have patience!

It's good to have some reference figures; thank you! (Also, welcome home!)

Sure thing. This was also early last week, I believe, when the traffic was slightly less. The thing that took the longest, as others have said, were the comments.

We're still holding at roughly 36 hours, so I think it's at least not getting worse. Knock on wood...

yes, i have to agree with a few others here that i appreciate you all being so communicative!
dumb questions: i assume we can still post entries while lj is being imported over here or will that cause a problem?

Edited 2017-04-10 11:45 (UTC)

No problem - post away!

oh wonderful, thanks! :)

I'm new here too. My LJ import a couple of days ago was in the queue for about 30 hours, but it came out fine in the end, with no errors. Thanks very much for offering this facility, even if I know you must be swamped. I hope to take out a paid account soon.

Good to have the numbers for reference -- thank you! (And welcome.)

Still waiting for mine to import - I'm guessing this is the busiest week you've had to date, so I'm not holding my breath. 12 hours down, about 24 to go by judging by the upper limit mentioned in comments so far (but judging by friend's updates in other media, LJ will be a wasteland soon enough).

Technically speaking I think there was more importing going on when we first opened to the public in May of 2009, although we haven't keep detailed traffic logs like that for eight years so I can't say for sure. You can sort of map out a timeline of bad decisions by looking at the detailed account creation history (newbyday), although those numbers can be a bit misleading: one new user creating an account and then importing their journal from LJ can cause dozens or even hundreds of new "accounts" (actually OpenID placeholders) to be created in order to properly attribute comments. So it's not that we had 25,000 individual people sign up yesterday. But it sure has been a lot! (There are more human-readable stats with pretty pictures, but the raw numbers give an interesting historical overview.)

AHA, I misunderstood - glad to understand this better!

My gmail folder where I store my account info tells me I've been on DW since 2010 and for most of that time (apart from the first year) I've been a paid member. Really, really impressed with the management - both their efficiency and their approach.

I had, or thought I had, a permanent (paid) account with LJ though I only ever cross post to it and use Scrapbook for hosting pics. But that hostage taking of my account seems illegal to me, as it clearly does to others, whether under US law, Russian, or any other. Still, I suppose people can't risk arguing or they could lose the lot.

Don't worry about being slow. We all understand. Most of us have friends and can explain to them if they're worried.

I'm excited about so many friends finally moving here. We need some kind of site-wide celebration in the near future!!

I made the joke about the block party after move-in, and then stopped and thought, huh, we really should have a site-wide party! DW's ninth anniversary is on 30 April, after all..

Leaving it to you clever coding people to sort out the 'how' of it all, but it really would be a lovely thing to do!

Actual brainstorming will have to wait until after we get through the initial OH MY GOD HELLO WELCOME LET ME FIGURE OUT WHICH OF THESE CARDBOARD BOXES ARE YOURS phase, though. :)

Good time to bring up

three_weeks_for_dw?

Always a good time to bring up three_weeks_for_dw!

erda still around?

Oh yeah, maybe a week or two from now would be a great time to bring that back. Or even just a "why not introduce yourself to the community" version of it maybe.

After being on LJ since December 2002, it felt weird to finally lower the boom and delete it entirely. But that new TOS? No. Just no.

So glad to have made a DW account when I did; thank you SO MUCH for the service to import our content!!

xoxo

You're welcome!

Same here. I imported my entries awhile ago and deleted my LJ a few days ago. I always thought I'd be on LJ in some way, shape, or form.

My journal is also newly imported. It took three days all told and the screen that showed the status of different elements of the import was very reassuring as the process progressed. It was easy to figure out that an awful lot of people were importing whole journals and communities at the same time, but it's especially nice to hear more of the details, such as the limits LJ sets in accessing. Thank you for creating DW and handling this massive influx.

Ooh! good to see you! I was about to thank you for subscribing to mine- it's locked however so you'll need to do the whole thing to get full access.

Did it now! :-)

Welcome aboard! It's good to know how long it's taken various people's imports to complete, too, so thank you. (I can see some details about how long a particular job's been running through the site admin tools, and if needs be I can poke

alierak or

mark to query the database directly, but we don't really have a way of saying "the average import job waited X hours in the queue before it started running and then took Y hours to complete once it started" or whatever, so it's good to have references.)

As for the arrival en masse of the last week -- the more the merrier, we say. People will figure out pretty quickly if they like the way we roll enough to stick around, but at the very least we can offer a stable backup platform.

And hooray for that!

We managed to get

holmestice into the queue around noontime on the first day, and it still took twenty hours all told. Last couple of times I checked queue length, the queue was three times as long (gauged by number of accounts waiting to import entries) as when Holmestice was in it. I haven't tried importing my personal journal yet, as I've been running

sanguinity and

sanguinity in parallel for years now, and figure I can wait until the rush dies down a bit.

I'm so pleased you did. I thought of all the fic and discussions that various communities contain and realised importing my own journal was not going to be enough.

I'm so relieved that the community approved of us backing the comm up to DW. All things equal, I prefer to ask before republishing people's works on a second site, but it did feel like there was some urgency.

Many people have been working hard this last week to import and archive things, here and on AO3 both, and I've seen discussions, too, about how to save the journals of deceased fans or now-abandoned communities. I'm not so optimistic as to believe we'll save everything, but I do hope we'll be able to minimize our losses.

Yes, I'm feeling hopeful, too, and appreciate the time moderators are taking to help it happen.

Thanks for all the hard work and for keeping us updated on progress!

I decided on Saturday to bring my old LJ over here for safekeeping (the entries are done, comments now in queue yay!) and had to laugh when I saw the length of said queues. Good grief, what an exodus! The housewarming party is going to be ginormous! *begins stockpiling munchies*

I'm really glad I'm not going to be responsible for paying the booze bill for the party, yeah. :)

(Long ago, when I was still working for LJ, we used to have a house party/retreat once a year for one of the volunteer teams: theoretically it was for us all to get together in person and have a week of serious-business meetings about how things were going, since the volunteers lived everywhere across the world. In practice, it was often an excuse for us all to get together, drink heavily, and play Katamari Damacy in a crowd. One year we held it in South Carolina, which had (has?) a law that you can't buy more than a certain amount of liquor at a time without filling out a lot of paperwork, in an attempt to keep you from buying in resale amounts: we had to buy that permit to transport all the booze we were buying back to the rental house from the liquor store. Twice. My liver is very glad I'm not still in my 20s, is all I'm sayin'.)

Here's hoping your comments arrive soon!

Ha! My liver agrees with yours. How we survived is beyond me. :)) That said, the SC house party sounds like it was pretty amazing.

Comments finished this morning - about 36 hours after dropping into the queue. Yay!

Now I'm importing a (very small) community and then I'll be able to delete LJ. Also yay.

Once this migration frenzy simmers down, I hope you'll update us on how many people moved in. The numbers must be impressive!

You can see them now! With fancy graphs, or the raw data -- you want "newbyday". (With the caveat that newbyday doesn't reflect how many people created accounts, just how many accounts were created; it counts the pseudo-accounts that are created as part of the import in that number, so one person creating an account and then importing their journal might be responsible for dozens or hundreds of "new accounts" if they had a lot of commenters who hadn't also commented in a journal someone had already imported. Like, if someone imports a journal I'd commented on with my LJ account, that wouldn't include an extra "account created", because the pseudo-account for my LJ account was created ages ago, but if someone imports a journal that has a comment by susiefromlj, and their journal is the only place susiefromlj ever commented, that would.)

IT IS A LITTLE BUSY, YES.

Oooh, that newbyday data is very interesting. I am intrigued with the spikes and now want to know what prompted them. May dig into history at some point to follow that thought. 2017 is going to be a banner year for Dreamwidth, it looks like. :)

Data point: my tiny comm import is progressing. Entries took 46 hours in queue, comments now queued. Given the insane queue numbers that seems pretty good to me! Thanks to you guys for keeping everything running under this onslaught.

Ha, yeah. You can kind of map LJ's bad decisions by spikes in our account creation as people import their stuff... :)

Thanks for all the hard work and for keeping us updated on progress! support i.

Thank you for letting us know! My journal entries are currently in the process of being imported from LJ. Is it possible to "subscribe" to journals that are on LiveJournal and to "grant access" to LJ friends through Dreamwidth? I tried to do this with several of my friends but haven't seen their journal entries on my "Reading" page here yet.

You can subscribe to the feeds from LiveJournal accounts and have them appear in your Reading page, but I think that will be public entries only.

When you import your LJ friends, they are given an OpenID to which you can grant access. In addition, if they also have a Dreamwidth account, they can claim their generated OpenID and link it to their Dreamwidth identity (if they want).

More information here: where to find LiveJournal feeds: http://www.livejournal.com/support/faq/149.html

All about subscribing to feeds, OpenIDs, etc
https://www.dreamwidth.org/support/faqbrowse?faqcat=othersites

I've declined to agree to the new LJ TOS having imported here years ago, so I am not logged in to LJ. I have one friend who hasn't moved here but who backs up in batches here. Even though we have mutual access in both places here I don't see her entries at all on my DW jounal. I used to just read her on LJ, but can't do that now. Can't do a feed from DW to DW. Help? No she doesn't want to just crosspost.

If I remember correctly, the imported entries won't show up on a reading page in order to not overload the reading page. Should show up on the actual journal, though.

Just lemme make sure I've got this straight: your friend posts her entries to LJ, then regularly imports them into her DW account, and you have her DW account in your circle?

If that's correct: Imported entries don't show on reading pages, mostly because if they did, I could (accidentally or deliberately) dump two thousand entries on your reading page just by importing and push off everything else that would otherwise be there. So unless your friend does switch to using Dreamwidth's native crossposter to post to LJ, you'd have to make a feed account for her public entries, and you won't be able to see her locked entries. (See me explaining that process here.)

Yes, you have it correct. Sigh. Oh well, at least most of the neighbourhood has now moved here!
Do you ever sleep? Just have to say that I REALLY appreciate you and Mark and the rest of the gang.

I alternate between not sleeping at all and sleeping way too much. (Ah, disabled life!) I have the worst case of Non-24-Hour Sleep/Wake Syndrome any of my doctors have ever seen; my personal 'day' runs anywhere from 20 to 40 hours based on any one of a thousand factors, some of which we've identified and some of which we haven't. It's one of the many reasons why I work for myself instead of for someone else, heh.

I don't think I've ever been up for more than about 30 hours but we stagehands don't have a very firm 24 hour clock either. Nothing like yours tho! Tomorrow I go to work at 6am and am loosely scheduled to get of at 2am, so a very long day.

Um, you say that imported stuff doesn't show up on the reading page? But, it does (at least for me). I see the imported entries of a comm of mine on my reading page. Or is it different with comms?

(the comm in question:

sga_news, this journal and, for example, this post.)

Yeah, communities are different -- entries in communities don't support the flag that we use for "don't show this on reading pages" (because of boring historical reasons). However, entries in communities also work in different ways in terms of "how to pick what gets shown on the reading page", so imported community entries older than two weeks old won't show on the reading page. (Whereas if we didn't flag personal entries that were imported, all of them would show.) When we introduced community imports, we looked at the problem for a bit, looked at each other, looked at the problem, and said "Eh. Good enough."

Ah! Thank you. I've just been confused because I knew from journals that imports won't show up on the reading page, imported the comm, saw the entries on my reading page and went *squeee* They managed to solve this. and then saw your comment here. XD

I'm just curious why personal entries would flood the reading page? Wouldn't they be alloted to the reading page at the "place" of the time stamp (you know what I mean?)? Or do they, in the background, get a "new" timestamp when they are imported (I'm not a programmer in any way or shape, so please excuse my naive questions.)? Because then I absolutely get why you won't let them to be added on the reading page.

I'm just curious because it works for the comms (even though you consider it a problem) and it would be such a cool thing for personal journals too.

Yeah, I know it doesn't seem like entries in comms and personal journals are any different, but "backstage", they handle time/date stamps very differently -- it's an old LJ quirk that carried forward to us. That's the "boring historical reasons" I mentioned! I would explain it, but it really is long and it really is boring to nonprogrammers.)

Anyway, we've never really fucked around with it because people doing imports is usually an infrequent thing -- it's not worth the development time it would take to retool everything, especially since "pick what goes on the reading page" is one of those very old code paths that's been optimized to hell and gone for performance and messing around with it is highly dangerous. (We tried to change something about how the reading page was rendered back in closed beta to eliminate that "if you're on the front page of your reading page, hit ?skip=20, and five people have posted since you loaded the front page of your reading page, 5 entries will be duplicated on ?skip=20 since the other 5 entries pushed them down", and promptly had three weeks of a heisenbug where some stuff wouldn't show on the reading page at all COMPLETELY AT RANDOM and we COULDN'T FIGURE OUT WHY. The reading page is deep magic with which it is not wise to fuck.)

Thank you for taking time and explaining this! :)

Of course! Always happy to explain.

We have to make that kind of cost/benefit calculation a lot -- the deep internals of the site are 18 years old (seriously), so even though they've got 18 years' worth of bugfixes and the like (which is why we didn't just start rewriting LJ from scratch), sometimes we decide it's a better idea to not poke at it too hard. (Icons are another thing like that. We once figured out that a thing some people were doing with their icons was entirely due to a bug and it shouldn't have ever worked in the first place...)

IIRC, Dreamwidth tried really hard for a long time to be able to offer locked posts in RSS feeds, but in the end LJ just would not grant them that level of access. I still appreciate that DW knew this would be something a lot of people wanted and tried their best to get it for us.

You've got a few answers already, but to cover exactly what you'll need to do: you can read public entries that your friends post on LJ on your Dreamwidth reading page via RSS ("feeds"). For instance, if you want to subscribe to

exampleusername, go to the Feeds page and put http://exampleusername.livejournal.com/data/rss in the "Feed URL" box. If someone else is already following that feed, you'll be prompted to subscribe to the account; otherwise it will ask you to give a name for the account, which can be anything, but will have _feed automatically added on the end of it. (Like, the feed of my old personal LJ account is named

synecdochic_feed, but when I created it, I just typed "synecdochic" in as the name instead of "synecdochic_feed"; the site put the _feed on the end automatically.)

Feeds will only show public posts. Unfortunately, there's no way to read someone's locked LJ posts on DW, even if you're on their LJ friends list; it's a limit of how the system works, and we weren't able to figure out a way to do it without being Evil or requesting data from LJ's servers way too often.

Quick warning: what appears on your reading page on DW from a feed is not the actual entry your friend posted. Don't leave comments on the 'entry' that shows on your reading page: follow the link to leave a comment on the original copy of the entry, not the feed version. Entries also don't stay in feed accounts for more than two weeks.

Thank you!
I have one more question. If i grant access to my Dreamwidth journal to my LiveJournal friends, will they be able to see my Friends Only posts? I hope so...

If you grant access to someone's LiveJournal OpenID (or let the importer import your friends list for you), your friends will be able to access DW using their LJ OpenID and read your locked posts without needing to make a separate DW account. They will still have to come to DW and authenticate with OpenID, though: there's no way for them to read your DW locked posts on LJ. (Other than you crossposting the posts to LJ, I mean, but that does involve sending your posts to LJ, and a lot of people don't want to do that anymore.)

Ooh thank you for waving this right in front of my face, I'm sure I knew it at some point but my memory was sure only paid users could create a feed!

...you know, you say that and I realize I don't actually know for sure if that's the case or not!

*consults her own damn site documentation*

Oh, good, I wasn't accidentally giving out bogus info after all; we really did make it so that anyone could create a feed. (You're probably thinking of LJ, where feed creation is a paid account benefit, and I think we had it that way for a while when we first opened before switching.)

I went straight away and tried it out by making a feed and it worked. I remember having to beg around for someone with a paid account to make me a feed on LJ.

(I do want to throw my money at you! But it's cool that we can do it for free!)

We will never look down upon people for not paying for an account! We specifically priced paid accounts so they'd subsidize free accounts at the paid account percentage-of-the-active-userbase we were predicting, and I'm happy to say that in eight years, those predictions have turned out to be pretty much on-the-nose. (Score one for my crystal ball.) I mean, we also won't ever say no to people giving us money! But don't feel guilty if you can't or just haven't gotten around to it yet.

I actually feel guilty over this! I had my import done early on, but then I decided to check my old old old entries and I realized that everything was a triplicate *headesk* That was a lot of entries to fix, so I deleted everything and started a new import??? Ops?

Virtual chocolate and cafeine to all!!!

Ahahahahah, yeah, oops, some people who did their first import way back when DW first started, like you did, get hit with a weird duplication bug when they import again now. (The joys of having to support eighteen years' worth of messy, real-world, edge-case data!) Sorry about that! Have a nice glass of your choice of beverage while you wait? :)

LOL yeah. Fixing entries in triplicate from 17 years ago was a drag tbh.

But my entries are done!!! :) Now to wait for the comments. This whole thing though is making me want to post again. That would be fun.

Thank you very much for your hard work

Не за что! (Which is about the extent of my Russian these days, even though I used to be much more fluent.) I'm glad that everyone's liking it here!

MY WORD, MONA, IT SURE HAS BEEN A WEEK, HASN'T IT

EVEN BY THIS ADMINISTRATION'S (basement level) STANDARDS YES IT HAS!

Flat | Top-Level Comments Only

Page 1 of 6

The importer is still running, we swear! It's just very busy.

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject