dw_maintenance | (no subject)

(no subject)

Update on yesterday's post about importing: we are continuing to carefully babysit the importer to ensure the maximum number of people's import attempts succeed, and we'll continue to add more resources as necessary in order to keep it from affecting usage for the rest of the site as much as possible. Thank you all very much for your patience!

If you've been waiting for a slower moment to retry an import attempt that failed: now is a good time to retry, since the overall queue is pretty short! If you started an import job earlier today or yesterday and you haven't gotten a failure in your inbox yet: there are a few comment import jobs that we're trying to baby along because they involve a very very high number of comments or because the comment threads in the account tend to be very deeply nested or involve very complex response trees. Because we preserve threading data on import, in order to keep the imported entry looking as much like the original entry as possible, it takes a lot of working memory to do a comment import for large accounts or accounts with very complex comment threads. We've been tweaking the settings on how much memory an individual import worker is allowed to take up: usually all that memory management is handled automatically, but we're trying to find the "sweet spot" for allowing complex comment imports to succeed without running out of memory, taking up all the memory and leaving none for any of the other import workers running, or losing the safeguards for not letting a single process get stuck and just eat up more and more memory over time.

alierak has been putting in heroic levels of work on managing the importer queue these past few days: thanks, Robby!

And, of course, some jobs are still failing because LJ is still intermittently blocking us, allegedly because "too many password failures makes a block happen automatically because it looks like someone trying to break into accounts". So there's still a chance your import will fail because it was assigned to a worker operating on an IP that's currently blocked. We're doing everything we can to work around that issue, but there's only so much we can mitigate it. If your import fails, wait half an hour and try again. (So far, the unluckiest person I've seen who kept hitting blocked workers took seven tries for it to finally work, but it did finally work!)

For people asking about Scrapbook photos: unfortunately, LJ doesn't provide a feed that would let us import those.

lannamichaels has a method to at least download your photos, and

blue_ant reminded me that Semagic, the Windows-based LJ client, will also download your photos (and notes that you may need to keep trying a few times).

Finally, because I've seen a bunch of people making references to deleting their LiveJournal account: before you delete your LJ account, it's a good idea to claim your LiveJournal OpenID with your Dreamwidth account. Doing this will cause all of your imported comments (including in communities), and any entries you made to a community that's imported, to update to having been made by your DW account instead of your LiveJournal OpenID account. Doing this before you delete your LiveJournal account will let you keep managing any old comments in your journal, any comments in a community that's imported, and any entries you made in a community that's imported, exactly as though you'd posted them with your DW account, and avoid the need to authenticate against your LJ account, which you can't do after it's been deleted. If you've already deleted your LJ account, we strongly recommend temporarily undeleting it and following that process before you delete it again!

EDIT: I forgot a lot of people don't know what OpenID is, sorry! OpenID is one of the protocols that lets you use one site's login credentials to log onto another site without having to create a whole separate account. If you've ever wanted to buy something from a website once, didn't want to create a whole account for it, and instead made the purchase with your Facebook/Google/AppleID login: that's the same concept. (Not exactly the same protocol, but the same concept.)

When the importer imports comments (or entries in communities), it attributes all the comments to the commenter's LiveJournal OpenID. That way, the person who left the comment can log into Dreamwidth using their LJ account and still have the same level of control they had over the comment (or community entry) as they had on the LiveJournal version of the entry.

Claiming your LJ OpenID account with your DW account means that when other people (or communities) import their journals, you'll be able to control those comments with your DW account instead of having to log in using your LiveJournal OpenID -- which you can't do anymore once you delete your LiveJournal account. It makes sure you don't accidentally lose control over comments and entries that you left in other accounts if those accounts have already been/are later imported to DW.

Flat | Top-Level Comments Only

I had a quick, clean import today (though it's possible that my old, original import already handled everything and there wasn't actually anything left to bring over).

Some suggestions that might help other folks:
- I logged into LiveJournal first, and they made me change my password because it was "too old", so that could be a failure point for some folks.
- Another could be use of the Google password manager, which might be defaulting folks' DW password in the import-kickoff pages; I just made sure to paste the new LJ one at each checkpoint.

Oh, good call, I forgot that LJ was doing the "change your password" thing. (Hilariously, except it isn't really hilarious at all, I just have to laugh because otherwise I'll scream: it's not because your password is 'too old', it's because your password was compromised in the massive LiveJournal security breach they have never admitted happened, much less disclosed to their users. If you used that same password anywhere else on the internet, for any other account, anywhere, go change it there, too: the dumpfile is now widely circulating and it's trivially easy for someone to get a copy of it.)

That might be better than my original suspicion -- that they force-expired passwords when they changed the TOS to be "Russian-language version is the only binding version". (Which I considered only because I hadn't signed into their site since they wanted me to accept that new TOS. Today I did, only to get access for long enough to export/delete.)

"Better" does not mean "remotely good", of course ...

Yeah, the "new ToS" thing they did was just making you log into the account and confirm that you agree to the new ToS; they didn't make you change your passwords for that. The forced password changes are because they couldn't keep pretending someone got their entire user database anymore after we disclosed it to our users because too many people had the same password for their LJ and DW account and we were going to have to force-reset passwords because bad guys were exploiting them for spamming purposes.

Yep, years ago LJ forced me to change my password and ever since I've been unable either to cross-post (I did change the LJ password on the DW crosspost settings to match, but it didn't help) or even to log in at LJ directly.

Through whatever weird loophole exists, (among other things they changed a hyphen in my username to an underscore) I've been (SO FAR) able to access my LJ Scrapbook by manually changing the hyphen that shows up on the URL to an underscore and then using the saved log in password on the Scrapbook page I keep as a perpetual tab.

I wish AO3 hosted images-- or at least that DW allowed linking DW hosted images to AO3 because fan art hosting is really fragile. After 3 times having to move to new hosts and update links in various AO3 works, I've given up on posting art at AO3.

The only thing that's stayed stable for me is the LJ Scrapbook, and right now that's not looking too good. First World Problems, I shouldn't complain.

Yeah, I wish we could afford to let people link to images from other sites too, but unfortunately that's really expensive -- as in, it can increase your bandwidth bill by several orders of magnitude. We just don't have the resources to do it! Image hosting is already one of the most expensive features we offer; that's why we have to restrict the quota so much.

Yes, that's what I thought. Maybe someday AO3 will be able to host images- if they restrict it to just the site, maybe they can afford it- I'm sure their budget is MUCH higher than DWs.

Eh, you'd be surprised! They have more assets on their annual Form 990 that they have to file with the IRS than we have, but that's because they buy servers and colocate them (and the server is thus counted as an asset and depreciates over time), while we use cloud hosting both for the ability it offers us to resize things on-the-go in response to a sudden demand and because we are done, done, done with having to drag ourselves out of bed and to the datacenter at 3AM in our pajamas to kick a recalcitrant server: that shit was bad enough in our 20s, we don't want to do it in our 40s, too. (The two factors are ... pretty equal in weight, heh.)

They also have a lot more overhead than we do: being a 501(c)(3) involves a lot of fundraising and fulfilment expenses, plus they pay for a lot more software than we do (they went 'pay someone to provide us these collaboration tools so we don't have to waste time building them', we went 'eh, we don't need to be that formal/organized, we can just use free software + chat + DW itself to keep track'), while we pay more people than they do (Mark and I take an extremely modest 'salary' monthly, and we pay Jen and Robby, our contractors, as well). There are a lot of differences between the two organizations! But in terms of pure "what each of us spends each year to keep the site available" without counting in any of those other variables, we're very much in the same ballpark. They have a lot more cash reserves than we do, we have way less overhead than they do. It all kinda balances out.

Thank you for all that very interesting information! :^)

You're welcome! It's basically two entirely different philosophies: they went for the nonprofit, donation-based model, while we considered nonprofit status, I took one look at the amount of paperwork it would involve every goddamn fucking year, and my ADHD-having, executive-function-lacking brain immediately noped RIGHT the fuck on out of there, heh. They went for owning the physical hardware they run on because they felt it gave them stronger protections against arbitrary disconnections by their upstream provider, we didn't think the risk of that happening with cloud hosting was significantly more likely enough to justify the extra work it causes and the way it sacrifices your ability to add more resources right this second if you get a spike in usage. They do a ton of (excellent!) other work in addition to offering the AO3, we don't have the energy (or the time -- I'm hella disabled, Mark has two young kids and a very demanding day job) to do anything more than just DW, and lord, I wish I had enough time to do everything I'd love to be able to do with DW, even, but these past few years have not been kind to me. (We finally got my spine fixed! The orthopedic surgeon who finally agree to fix it had some choice words for the orthopedists who blew me off for four years' worth of me steadily losing more and more of how much I could sit up per day until by the time I got in to him, I was at "if I am very very lucky I can get out of bed long enough to eat and pee" level of bedbound. It was a 3 hour surgery with zero complications, I was out of the hospital in 48 hours, and 3 weeks post-surgery I was completely recovered: now I am just having to rehab all the muscle tone I lost in four years of unnecessary pain and immobility, sigh.)

Anyway, I digress! They're two entirely separate approaches, and you can kinda boil it down to: the AO3 went for the "very spread out, loads of people, governance by democratically elected board, build the organization with the extreme long view" approach, we went with the "minimal overhead, less paperwork, retaining control over the decision-making process so we don't ever risk someone who doesn't agree with our vision being the one making decisions, centralize decision-making power but show people they can trust us by being transparent in our decision-making process" approach. Neither is better or worse than the other! Both of them have their advantages and disadvantages. (For example, they tend to get bogged down in committee and debate about things forever because there's unclear structure on who has the authority to be the final word on whatever they're debating; we are really susceptible to "people who are working on a thing don't want to bother the people with the decision-making authority, aka me and Mark, because they know we're busy/overstressed/in a lot of pain right now, so things sit forever until one of us is available".)

I am getting far afield from my original point, heh. Which was: AO3 has much higher cash reserves than we do -- we aim to keep 5-6 months of expenses on hand in cash and another few months' worth in available credit limit; the credit line stays reasonably stable while the cash reserves fluctuate slightly, mostly based on how long it's been since we've had to write a check to the IRS -- but they also have much higher overhead and would have a much higher necessary capital outlay if they decided to move into image/media hosting.

From what I understand, they also have a lot of trouble finding qualified people with the necessary experience in server ops and site availability, because it's the kind of shit job nobody really wants to do on a volunteer basis unless they really, really, really believe in the cause, and because you don't want to give a newcomer access to all your everything (which server ops inherently involves) it takes a lot of time to build trust with new people. Whereas we have the advantage that Mark's entire career has been devoted to that kind of thing (you definitely have heard of the place he works for his dayjob) and Mark has known Robby and Jen since they were all just barely out of their teens, so trust isn't a problem. Again, different approaches, and they all pretty much balance out.

Hosting media is hard, expensive, annoying and fiddly to keep running, and also (in the US) opens you up to a lot of potential legal issues that text alone doesn't carry: they may have decided not to do it (or decided not to do it yet) purely on that basis alone. If I hadn't been working in the field for literally 20 years I would be way more nervous about making the necessary judgement calls myself, honestly.

Thank heaven you finally got a doctor who helped you. *HUGS*

I really appreciate all this insight - so nice for ME to reap the benefit without having done any of the work. I collect random information about real people like a magpie going after shiny bits, and all this is new to me and so really fascinating.

As long-term residents know, the trick isn't getting me to explain something, the trick is to get me to stop explaining before I run out of comment limit, heh. (True story: the first thing I made Mark do was expand our comment limit like, significantly over LJ's, because I was tired of having to post 5 comments in a row to explain something!)

Hah! It's hard to stop once you're on a roll. :^)

I LIKE explaining things! There's so much NUANCE to explore! wait where are you all going this 15,000 word essay discussing the history of legislation of the internet and how it created today's social media landscape is really interesting I promise

Edited (hit enter too soon!) 2022-03-12 21:58 (UTC)

*gets a drink of water, pulls up a comfy chair and prepares to take notes* :^)

All it needs is a cat... oh, wait. The internet was built on LOL cats, so I'm sure they're in here somewhere. :^)

Please? Pretty please? (I mean, when you have time and things, whenever that might be.)

Gods but I missed getting to see these kinds of comments over the last 4 years, dear one!

<3

Flat | Top-Level Comments Only