import queue delays

[denise]

Thanks to a few people reporting problems with their import jobs stuck on the "verify" step for an extended period of time, we discovered a bottleneck in the import process we hadn't realized. We've taken steps to fix it. If your import was showing as "ready to be inserted into the queue", those jobs are now being moved into the import queue more quickly. (That's why, if you've been watching the queue on the Import Journal page, the numbers just jumped like whoa.)

It will take time for the importer to process all the queued jobs -- whenever there's a surge in account creation, there's a corresponding surge in import jobs -- but fear not, once they're scheduled your import jobs will run. You don't have to leave the page open: just schedule the job and wander off, and sooner or later you will look at your journal and all of your stuff will be there like magic. :)



Basically, the problem was: we actually have two import queues. The first is the queue for the "import-scheduler" job: it verifies your username and password on the remote site (since there's no sense in retrying a job that's going to fail because the authentication is incorrect) and then puts the job into the scheduler queue. From there, a worker moves the job from the scheduler queue to the actual import queue for a TheSchwartz worker to actually do the import. (The reason why our worker-manager is known as TheSchwartz is a long, long story. *G*) The import queue that was showing on the import page is the TheSchwartz import queue, not the import-scheduler queue.

We discovered, when people reported problems with the "ready to be inserted into queue" issue, that the job that moves jobs from the import-scheduler queue to the TheSchwartz import queue was set to only move one job from queue to queue every 60 seconds. This is a delay we built into the system deliberately, and frankly, none of us remembered quite why -- it was either to avoid overwhelming things and was there from the beginning or was a kind of artificial delay that we put in during a period of LJ DDoS. Usually it's not a problem, because very few people are trying to import at any given time and both queues are usually at or close to 0.

So, as it turns out, that "one job every 60 seconds" combined with the high import traffic today meant that there were over 1200 jobs in the import-scheduler queue, being moved to the TheSchwartz import queue very very slowly while more and more came in. Hence the backlog!

We've removed the artificial delay, and jobs are now being moved from the import-scheduler queue into the TheSchwartz import queue as they come in and can be verified. So, the only limit now will be the speed at which the imports can run.

EDIT, 8:40PM EDT: Sorry about the rampant internal server error problems -- we thought it was a problem with the new webserver, but it turned out that imports were happening too fast and were locking up the database. Mark has throttled back the import speed enough that the errors should go away now. (This means that imports will be happening more slowly, but the queue's backed up enough right now that it probably won't make much difference anyway!)

EDIT, 4:30 PM EDT, 12/23: As always happens whenever we have an influx of new users, the import queue is very, very busy right now. Your import will almost certainly take at least a day to finish. Please be patient! Once your job is in the queue, it will complete eventually and you don't need to stay logged into the site or leave your computer on. Just start it and go do other things, and eventually your stuff will catch up with you. :)

Comments

[denise]

oh god yes. It's somehow worse when you're looking at your own code, too, because you are convinced you should know what the fuck you were thinking, and you really, really don't.

[owl]

Oh hell yes. I'm not sure which is worse: "This file has my @author tag on it and I don't remember a thing about writing it", or "I wrote this the day we went to the market and M refused point-blank to have any lettuce on his burger, and we talked about this crazy logic - but I can't remember the reason."

Or when there are comments and they go like This is a kludge to get around the bletcherous error-handling API of $OTHER_DEPARTMENT or Remove this method whenever $THIRD_PARTY_VENDOR goes away - and you're reading the code three years later and $OTHER_DEPARTMENT's code is still bletcherous and $THIRD_PARTY_VENDOR is still going strong.

[denise]

oh god yes. There are a few bits in DW code that have comments like "blame this on mischa" AND THAT'S ALL. It's really a good thing that either mark or I usually remembers what was up ...

[azurelunatic]

Thank goodness for you and Mark!

... though I do have to ask, is that Mischa in his coder hat, or in his black hat?

[denise]

heh, I was thinking the coder hat :)

[azurelunatic]

The latter of the former, sometimes if you remember the day, you can pull up the journal entry, and the journal entry may have explained the logic. If you're lucky.

[archangelbeth]

DW's ability to search one's journal was really useful for when I had to figure out the date that my kid had such-and-such operation. (I think it was the one to sew her bellybutton back on...) So sez the non-coder. O;D

[denise]

Talk about your uses you didn't expect!

[archangelbeth]

It was extremely useful indeed! O:D

[reddragdiva]

I use DW as pretty much my actual diary and have used it as a useful reference.

[susanreads]

The second para of this comment made me LOL for real.
- former programmer
(There are parts of having a job that I miss, not just the having-an-income part, but there are parts I don't)