Dreamwidth Maintenance

Someone reminded me we hadn't made a top-level post about the "backend fetch error" that people have been getting in the evenings -- this is a problem related to site load. (It's also related to the increase in abusive traffic we've been seeing over the last several months or so, and we're not the only ones who've been noticing it; there's been a general internet-wide upswing in garbage lately.)

We're doing our very best to keep legitimate site usage from noticing the problem. When you get any kind of site capacity related error, wait a few minutes and try again: our systems have a number of ways to automatically fix the problems, but it takes a few minutes for them to get more capacity online. Sometimes it can also take a few rounds of "add more capacity, see if that fixes the problem" for the issue to fully resolve. We've been looking into long-term fixes that we can make to reduce the amount of garbage traffic we're getting and the chance that it will overwhelm our capacity, but we're already getting feedback from legitimate site users that the traffic filtering steps we're taking are causing issues accessing the site for them, and we're trying to avoid making that problem worse.

We're really sorry about the hassle! Things may be a little rocky in the evenings US time for the next week or two as we work to find the best long-term solutions, because evening in the US is both a higher period of legitimate activity for us (because of our high number of US users) and a higher period of abusive traffic (because it's the beginning of the day in the countries we get the most abusive traffic from). If you do get an error, waiting 3-5 minutes and trying again should resolve it.

EDIT: This issue has now been resolved and all of our support email aliases are working again. We still don't know how long they were down, and as I suspected, all of the email that was sent to those aliases while they were down can't be redelivered. If you tried to contact us using one of our support email aliases and you haven't heard from us, please send the message again: chances are very good we never got it. I'm really sorry for the hassle!

Original message about the now-resolved situation follows: ( No longer applicable announcement )

We're investigating reports of problems with payments not properly being applied to accounts, and will have it fixed (and all the payments credited) as soon as we possibly can!

EDIT (11:50PM EDT): We've found the root cause of the problem: over the past few days we've been having a heavy uptick in automated traffic that's been causing some slowdowns, and put a few steps into place to block that malicious traffic. It looks like we also accidentally caught the IP address that our payment processor uses to notify our backend of payments in that block range! Totally our mistake, we apologize profusely, and any payment that hasn't properly been applied to your account should be processed within the next hour or two.

We have managed to chug through every single outstanding "things broke because there were too many comments/something in the data feed was busted/we have no idea why it broke but it runs okay when we run it manually" import that people alerted us to, and the import queue has been holding steadily at 0 for the last 24 hours (ie, jobs are finishing pretty much as soon as they're scheduled), so we've cut back the extra resources we diverted to the importer for this week. If you noticed a brief blip (like 30 seconds) of the site giving 502 errors, that was us fiddling with the memory allocation and the databases again!

We thought we'd figured out the underlying cause of why imports with a large number of comments were failing, but every time we've thought we solved the thing that would fix it, it's turned out the actual issue is down one layer of abstraction. We're still working on figuring that one out. Yes, it's driving us up the wall. We're sorry! We will let you know when people can do large imports again. (If it turns out after we find and fix whatever bug is causing the problem that we also need extra memory for those to succeed, we'll throw that extra memory at it for another few days at that point so people can have another window to import their larger accounts without the job running out of memory -- we just have to turn it back down for a while because it is more expensive to provision that much memory and we don't have the budget to do so indefinitely, especially if it's not being used like it isn't right now. But a few days is not a problem.)

Thank you again to everyone who's been so patient with us this week. We really, really appreciate it.

We believe we've figured out the underlying reason why larger import jobs (300,000+ comments) are failing! Right now we're manually running a few larger jobs to verify that our current theory is correct (it's a convergence of several different factors), and if we are correct, we'll put in a quick and dirty fix so those jobs will no longer need manual intervention. In the meantime, if your account or community has more than 300,000 comments, hold off on restarting another import: we'll let you know when you can try again.

Update on yesterday's post about importing: we are continuing to carefully babysit the importer to ensure the maximum number of people's import attempts succeed, and we'll continue to add more resources as necessary in order to keep it from affecting usage for the rest of the site as much as possible. Thank you all very much for your patience!

If you've been waiting for a slower moment to retry an import attempt that failed: now is a good time to retry, since the overall queue is pretty short! If you started an import job earlier today or yesterday and you haven't gotten a failure in your inbox yet: there are a few comment import jobs that we're trying to baby along because they involve a very very high number of comments or because the comment threads in the account tend to be very deeply nested or involve very complex response trees. Because we preserve threading data on import, in order to keep the imported entry looking as much like the original entry as possible, it takes a lot of working memory to do a comment import for large accounts or accounts with very complex comment threads. We've been tweaking the settings on how much memory an individual import worker is allowed to take up: usually all that memory management is handled automatically, but we're trying to find the "sweet spot" for allowing complex comment imports to succeed without running out of memory, taking up all the memory and leaving none for any of the other import workers running, or losing the safeguards for not letting a single process get stuck and just eat up more and more memory over time. alierak has been putting in heroic levels of work on managing the importer queue these past few days: thanks, Robby!

And, of course, some jobs are still failing because LJ is still intermittently blocking us, allegedly because "too many password failures makes a block happen automatically because it looks like someone trying to break into accounts". So there's still a chance your import will fail because it was assigned to a worker operating on an IP that's currently blocked. We're doing everything we can to work around that issue, but there's only so much we can mitigate it. If your import fails, wait half an hour and try again. (So far, the unluckiest person I've seen who kept hitting blocked workers took seven tries for it to finally work, but it did finally work!)

For people asking about Scrapbook photos: unfortunately, LJ doesn't provide a feed that would let us import those. lannamichaels has a method to at least download your photos, and blue_ant reminded me that Semagic, the Windows-based LJ client, will also download your photos (and notes that you may need to keep trying a few times).

Finally, because I've seen a bunch of people making references to deleting their LiveJournal account: before you delete your LJ account, it's a good idea to claim your LiveJournal OpenID with your Dreamwidth account. Doing this will cause all of your imported comments (including in communities), and any entries you made to a community that's imported, to update to having been made by your DW account instead of your LiveJournal OpenID account. Doing this before you delete your LiveJournal account will let you keep managing any old comments in your journal, any comments in a community that's imported, and any entries you made in a community that's imported, exactly as though you'd posted them with your DW account, and avoid the need to authenticate against your LJ account, which you can't do after it's been deleted. If you've already deleted your LJ account, we strongly recommend temporarily undeleting it and following that process before you delete it again!

EDIT: I forgot a lot of people don't know what OpenID is, sorry! OpenID is one of the protocols that lets you use one site's login credentials to log onto another site without having to create a whole separate account. If you've ever wanted to buy something from a website once, didn't want to create a whole account for it, and instead made the purchase with your Facebook/Google/AppleID login: that's the same concept. (Not exactly the same protocol, but the same concept.)

When the importer imports comments (or entries in communities), it attributes all the comments to the commenter's LiveJournal OpenID. That way, the person who left the comment can log into Dreamwidth using their LJ account and still have the same level of control they had over the comment (or community entry) as they had on the LiveJournal version of the entry.

Claiming your LJ OpenID account with your DW account means that when other people (or communities) import their journals, you'll be able to control those comments with your DW account instead of having to log in using your LiveJournal OpenID -- which you can't do anymore once you delete your LiveJournal account. It makes sure you don't accidentally lose control over comments and entries that you left in other accounts if those accounts have already been/are later imported to DW.