slowness/downtime

[mark]

Hi all,

In the past 24 hours we've had some periods of slowness and outright downtime. fu and I have managed to track it down to a single community that has a comment thread that causes our Apache workers to go into an infinite loop. After enough refreshes, all of our workers are dead, and the site no longer responds.

We're going to start looking into fixing the code so that this doesn't happen. Meanwhile, we're going to work with this community to make sure that they can't break the site while we get an actual, proper code fix working.

Thank you all for your patience as we sort this out.

Comments

[poulpette]

Thank you for your hard work fu and mark!

I'm curious as too how a small comm manages to hog all the DW plates though. That seems like an odd one :D

[azurelunatic]

My personal guess: a roleplaying game or something with very, very long comment threads. From the comments I see we are likely to get a description of the type of thing it was after there is a permanent fix, yay!

[mark]

Yes --

It was a small RP community. They had gotten a post up to 1400 comments, which is really nothing, but they were arranged in a chain. Reply, reply, reply, reply. So instead of the normal view of things -- a wide tree -- we had a very, very deep tree.

The part of the code that checks to see if you are subscribed to a thread walks the tree from where you are up to the top to see if you're subscribed to any comment in the chain. Normally, this is simple -- you walk a few comments up and you're at the root and you're done.

In this case, it had to walk 1400 comments. The code was done in such a way that it had to do this frequently, and it caused an explosion of work on Apache, taking tons of memory and CPU. Ouch.

fu fixed it temporarily with some better caching to make sure we don't have to recompute things we already know. We may come up with a better way to do it, but it works for now.

Aaaaaaaanyway. I know I didn't break that down into plate analogies, but I have to run. Maybe someone else can, if needed, and if not, I'll do it later if I remember. ^^;

[denise]

Oooh, oooh, I'll try!

So, whenever you and your friends go out to dinner, you all share your food by picking up your plates and handing them around the table so everyone can get a taste. When you've got four people at the table, that's no problem, you can make one circuit of the table really quickly and finish your meal and the plates can get back to the kitchen to get washed and reused.

But, say you've got a really big party. And everyone wants a taste of everyone's dinner still. So, you pass the plates around the table just like you're used to doing, but since there are so many of you, it takes the plate forever to make its way around, and by the time it's only halfway around your group, the restaurant says, "Um, guys, can you speed this up? We totally need the plates."

[azurelunatic]

Ooo, I like yours.

[azurelunatic]

Dodgy analogies are a specialty of mine!

So there's this person building a staircase who needs to be served as well. One of the workers runs up a plate to the guy who is at the top of the staircase, going step by step up. It takes far longer for that plate to get there and back again than for someone in the main dining hall, and then it might not even get washed. And the staircase keeps increasing in length, but the guy on the staircase still needs to get served, so it takes longer and longer and the person doing the plate-running starts to get very tired out.

This is a situation for a bucket on a pulley.

[andy]

So the loop isn't really infinite, just a very long one, fair enough. :)

[mark]

Yeah. When I made the original post, I thought it was an infinite loop. It turns out it's not, it's just a really bad tree traversal problem.