import queue delays

[denise]

Thanks to a few people reporting problems with their import jobs stuck on the "verify" step for an extended period of time, we discovered a bottleneck in the import process we hadn't realized. We've taken steps to fix it. If your import was showing as "ready to be inserted into the queue", those jobs are now being moved into the import queue more quickly. (That's why, if you've been watching the queue on the Import Journal page, the numbers just jumped like whoa.)

It will take time for the importer to process all the queued jobs -- whenever there's a surge in account creation, there's a corresponding surge in import jobs -- but fear not, once they're scheduled your import jobs will run. You don't have to leave the page open: just schedule the job and wander off, and sooner or later you will look at your journal and all of your stuff will be there like magic. :)



Basically, the problem was: we actually have two import queues. The first is the queue for the "import-scheduler" job: it verifies your username and password on the remote site (since there's no sense in retrying a job that's going to fail because the authentication is incorrect) and then puts the job into the scheduler queue. From there, a worker moves the job from the scheduler queue to the actual import queue for a TheSchwartz worker to actually do the import. (The reason why our worker-manager is known as TheSchwartz is a long, long story. *G*) The import queue that was showing on the import page is the TheSchwartz import queue, not the import-scheduler queue.

We discovered, when people reported problems with the "ready to be inserted into queue" issue, that the job that moves jobs from the import-scheduler queue to the TheSchwartz import queue was set to only move one job from queue to queue every 60 seconds. This is a delay we built into the system deliberately, and frankly, none of us remembered quite why -- it was either to avoid overwhelming things and was there from the beginning or was a kind of artificial delay that we put in during a period of LJ DDoS. Usually it's not a problem, because very few people are trying to import at any given time and both queues are usually at or close to 0.

So, as it turns out, that "one job every 60 seconds" combined with the high import traffic today meant that there were over 1200 jobs in the import-scheduler queue, being moved to the TheSchwartz import queue very very slowly while more and more came in. Hence the backlog!

We've removed the artificial delay, and jobs are now being moved from the import-scheduler queue into the TheSchwartz import queue as they come in and can be verified. So, the only limit now will be the speed at which the imports can run.

EDIT, 8:40PM EDT: Sorry about the rampant internal server error problems -- we thought it was a problem with the new webserver, but it turned out that imports were happening too fast and were locking up the database. Mark has throttled back the import speed enough that the errors should go away now. (This means that imports will be happening more slowly, but the queue's backed up enough right now that it probably won't make much difference anyway!)

EDIT, 4:30 PM EDT, 12/23: As always happens whenever we have an influx of new users, the import queue is very, very busy right now. Your import will almost certainly take at least a day to finish. Please be patient! Once your job is in the queue, it will complete eventually and you don't need to stay logged into the site or leave your computer on. Just start it and go do other things, and eventually your stuff will catch up with you. :)

Comments

[moem]

Thanks as always! I was wondering, but decided to wait it out, and behold! You've already fixed it!

<3 <3 <3

[blackmare]

Thanks for this. I linked it for someone who was worried about the lag time on their import.

[amai_kaminari]

You're awesome, DW staff! <3

[tabaqui]

Yay, thank you!

[instantramen]

Whenever I see one of you mention TheSchwartz I assume it's a Spaceballs reference. Use the Schwartz!

[denise]

It is actually because the LJ director of engineering at the time -- aka the manager of the people who wrote the worker manager, including mark -- was named Mike Schwartz, but I'm positive it was also a Spaceballs reference.

[icelightning]

Thanks for the quick solution as always!

[geminianeyes]

Have I mentioned that you guys are awesome? :D

[ironed_orchid]

You are awesome.

[backtothelight]

This kind of honesty and transparency is honestly new to me, as I am a longtime LJ user. I am very impressed. Thank you guys so much.

[sophie]

The honesty and transparency here is awesome. Have you seen the code tours yet? They're basically a thing done once every few weeks that looks at all the bugs that have been fixed since the previous code tour and explains every single one in an understandable fashion. (The fixes normally aren't live on the site when a code tour is done, but they become live on the next code push.)

[soc_puppet]

One more reason y'all are made of awesomesauce ♥

[jelazakazone]

You guys are like magic.

[wldcatsprstr_14]

Fancy meeting you here...

[caeseria]

I'm getting a whole boatload of server errors whenever I try to do something at the moment...well, for the last couple of minutes anyway. Is there any connection?

Thanks.

[denise]

Looking into it! web03, the server we added today, is having some issues; we've temporarily taken it out of the pool until we can figure out what's up.

[ashoakandthorn]

All I have to say is this: Thanks for all the work you've put into this site!

It's wonderful, and I always know you'll have our backs and that if there is anything that crops up, I feel like I can come forward and it is actually address.

That said, I had been getting error messages (code 500)too, but they've gone away, whatever you happened to have done worked.

So thank you, thank you, thank you.

[skinners_box]

As part of the mob that is moving over from LJ, I have to say sorry for the mess, but thank you for the responsiveness! This account has already imported and it looks great!

[got_swagger]

You guys are the absolute best! I love how dedicated to being honest and transparent with your users you are!

[swankyfunk]

Hi, I'm currently importing over 6k entries from my LJ, so it's understandable that it's taking a really long time. But I don't have to stay logged in here, do I? Just wondering. Thanks!

[sophie]

Nope! You can close the window and/or log out and it'll still keep chugging along.

Thank you!

[quicksilver_ink]

I haven't been importing entries, but I just wanted to say how awesome it is that you guys are transparent and listen to your userbase. After the latest debacle at LJ, I really appreciate the difference.

[owl]

This is a delay we built into the system deliberately, and frankly, none of us remembered quite why

I'm going to guess this is a config setting and not code (or else the commit comment says "Re v 435 - Frobnicated the bobbits")

[denise]

*g* We figured it out after I posted -- it was there from the start, and it was a holdover from the initial design of the import system, in order to prevent things from getting too overloaded, just as a failsafe. Turned out we'd since changed how the worker runs and exits so the thing we were designing around didn't apply anymore!

[die_monster]

you guys. YOU GUYS. <33!

[schexyschteve]

Wow, you all are incredible. I'm getting warm, fuzzy feelings from this site, and I'm brand new here. I like it. :D

Look at the communication with everyone! It's so pretty!

[moem]

Nice, huh. Welcome to Dreamwidth!

[skinners_box]

So I saw on Twitter that there's a pending tool for comm imports. Is this the comm I should watch for updates on that? I'm waiting with baited breath.

[denise]

dw_news :)

[vickyblueeyez]

What's the link to see the numbers to the import queue? Also, some LJers are still under the impression that you can import communities. Is this true? Maybe I didn't see the news or something.

[denise]

http://www.dreamwidth.org/tools/importer

Mark's been working on community import for a bit (the limiting factor was OpenID accounts being able to post to comms so people can still maintain control over their stuff like they can with comment imports). Hopefully by the end of the week, if he can hack around the baby. :)

[rezykel]

Forgive me if this is written elsewhere but I couldn't find it: when you import all the things from LJ, are the security settings on the individual posts also preserved? (Like, if a post was set to Filter X at LJ, it won't suddenly become public or Filter B?)

[denise]

Your filters are absolutely preserved, don't worry! When you choose the import, if you also choose to import 'friendgroups', it will create the same access filters on DW and populate them with the OpenIDs of the people who were in that filter (you'll have to add their DW accounts manually if you know they have one; we haven't gotten to the feature that will let people associate their OpenIDs with their DW accounts yet). If you don't import friendgroups, I can't remember if it still creates the access filters (and just doesn't populate them) or if it sets the posts to be locked to a non-existent filter, thus functionally private. I think it creates any access filter that was used for posting to, but I'm not sure. Either way, the system will make sure things fall back to less public, not more public.

[clawdine]

Hi DW! Using the link provided on the profile page for DW username no longer works on LJ entries! I'm thinking this is asshattery on LJ's part, but just to let you know.

[denise]

Thanks! can you give me a link to an entry or comment on LJ that shows the problem in action? (Seeing how it's failing will help to diagnose, etc.)

[bell]

Gotta say, the past few days of being stonewalled by LJ reminds me just how much DW's clarity of communication alone makes it worth being here.

[denise]

Thanks for the compliment!

Just to let you know (and I'm replying to everybody in the comments here who's made similar comments, so please please don't think it's personal!) -- we're asking that people not make negative comments about LJ in the comments to our official communities, since we just don't think it's right to be encouraging that in our official spaces, especially since we literally could not exist without LJ having existed first and many people already have a negative impression of evangelism by people who support DW. I know emotions are running high and everybody's feeling very stirred up right now, and we don't want to make things worse in our official spaces!