Friday import update

[denise]

We have managed to chug through every single outstanding "things broke because there were too many comments/something in the data feed was busted/we have no idea why it broke but it runs okay when we run it manually" import that people alerted us to, and the import queue has been holding steadily at 0 for the last 24 hours (ie, jobs are finishing pretty much as soon as they're scheduled), so we've cut back the extra resources we diverted to the importer for this week. If you noticed a brief blip (like 30 seconds) of the site giving 502 errors, that was us fiddling with the memory allocation and the databases again!

We thought we'd figured out the underlying cause of why imports with a large number of comments were failing, but every time we've thought we solved the thing that would fix it, it's turned out the actual issue is down one layer of abstraction. We're still working on figuring that one out. Yes, it's driving us up the wall. We're sorry! We will let you know when people can do large imports again. (If it turns out after we find and fix whatever bug is causing the problem that we also need extra memory for those to succeed, we'll throw that extra memory at it for another few days at that point so people can have another window to import their larger accounts without the job running out of memory -- we just have to turn it back down for a while because it is more expensive to provision that much memory and we don't have the budget to do so indefinitely, especially if it's not being used like it isn't right now. But a few days is not a problem.)

Thank you again to everyone who's been so patient with us this week. We really, really appreciate it.