site slowness: it's complicated!

[denise]

The site slowness for the past day or so has been due to a bug somewhere in our code that's causing our webserver processes to run out of memory too quickly and lock up the machine.

alierak has been staying on top of things and tweaking the webserver settings to keep things running and to make sure that the settings we're using have the best chance of not running into the "run out of memory, lock up machine" problem. Unfortunately, this means that -- in order to minimize the chance that the site is down entirely -- we've had to seriously lower the number of webserver processes that are running at any time and lower the amount of time before they restart by themselves (and free up the locked-up memory). This means that there are fewer webserver processes available to accept your requests and serve you pages from the site.

Basically, at this point it's a case of "down because of the problem or slow because of the steps we're taking to fix the problem"!

Since it's obvious at this point that just webserver tweaks isn't going to cut it for now, we're doing two things to get the site back to its usual zippy self:

a) Trying to find the root cause of the bug that's making our webserver processes freak out. Memory leaks are really hard to find and debug, which is why it's taking so long. We have a few ideas on how to find what's causing it, and fu is concentrating on that end.

b) Seeing what we can do to get more resources into the webserver pool so that even though the webservers are running out of memory quickly and we have to resource-starve them in order to keep them from checking out entirely, we'll still be able to get pages to load quickly without the delay we're experiencing right now. There's an easy way and a hard way for this, too. (And hopefully, the easy way will help enough that we won't have to get to the hard way.)

(This sort of thing always happens when mark is literally unreachable -- he's on vacation for two weeks in remotest Alaska, with no cell phone reception -- but I wanted to specifically give a massive thank you to alierak, our backup sysadmin, who is doing wonders with the problem.)

Comments

[halialkers]

It's always good to see that you guys are on the ball and giving timely updates. ^.^

[spinpsychle]

Don't forget accurate!

[dickgrayson]

Good to know! Thank you! :D

[hellkitty]

Thanks for keeping us in the loop! We'll be patient.

[girl_76]

Thanks for the update. I also imagine people jumping ship from LJ to here is slowing things down a bit.

[helens78]

Great big props to alierak, and thanks for keeping us posted!

[turlough]

Thank you for the update!

[starwatcher]

.
Thanks for letting us know. I got a virus a while back, that I'm not sure isn't still lurking, and tend to worry that any slowness is confirmation of a hidden virus. It's good to know there are other causes.
.

[bell]

Go team! :)

[elfin]

Thanks for keeping up updated. <3

And good luck with the bug hunting!

[jumpuphigh]

Thanks, D, for the update. I know it must be stressful but we support you and are willing to be patient.

(Also, my computer does the same thing with memory. It's a pain.)

[eleanorjane]

Thankyou, especially for the awesome transparency. The updates are much appreciated.

[kitters]

I love you guys so much. ;_; And I love whoever randomly decided to gift me with a month of paid time.

Ah. Good to know.

[hillbillycavewoman]

I thought it was me.

[sweetmeow]

Wondering if part of the issue is that Live Journal is down - and there is an extra strain on resources because more are over here. Add that to the code issue that you describe, and that would make it worse. Is that possible?

[denise]

It's not so much that the extra load is causing problems directly, as that the memory leak only seems to happen under load. Usually when we get a rush of traffic like this we can handle it just fine, but in this case it seems to be what's triggering the memory leak problems and causing the webservers to lock up. So it's not that the extra traffic is the cause of the problem, but that the extra traffic is triggering the bug that's causing the problem!

[alisanne]

Thanks for the update! It's good to know I'm not imagining things. :)

[sharpiefan]

Thanks for letting us know! Here's hoping you get it sorted soon!!

[aedifica]

Thanks for the news, and I wish you each your favorite caffeinated beverage while you're working on this. (Or non-caffeinated, for anyone who's staying away from caffeine--it just seems like the kind of situation that calls for it.)

DW is love right now.

[wytchcroft]

a bug? i hope not a deliberate one - coz the timing with lj is just nasty.
you guys rock, do what you can and i think you'll find that most folks here are patient - i certainly don't envy you! *Hands out sympathetic coffee and Danish.*

Re: DW is love right now.

[azurelunatic]

Some terminology nuance -- 'bug' tends to be a (known or unknown) unintentionally created flaw that causes a problem, either when something's wrong by itself, or two or more things that would be OK on their own come together and create a problem -- like, there's nothing inherently bad about a toddler, and nothing inherently bad about an unlidded jar of peanut butter, but put the two together and there's a problem, especially when the cat then walks by.

Then there's a whole class of jargon for security problems -- there are bugs that are also 'vulnerabilities', weak points that could be attacked by someone. Sometimes the vulnerabilities will be exploited, which was what I think you were hoping wasn't happening.

*brews some more sympathetic coffee, finds some Diet Coke with lime for denise*

[evilawyer]

Thanks for keeping us updated. (Would it help any if people went a little slower on posting or avoided posting at peak times? Don't know how this things work, but willing to lay off the posting and replying if it will help.)

[denise]

There isn't really anything that one person can do to help out! (But thank you for asking. *g*) The problem isn't the traffic itself, it's that there's a bug causing the webserver to lock up that we wouldn't have noticed without the load. (At our usual traffic patterns, the normal mechanisms to 'restart' the webservers regularly was enough to keep the webservers from locking, so we didn't notice the problem until the past few days.)

The best thing you can do to help is just be patient with the slowness! We're juggling a few things around in the hopes that we can make the site a bit faster while we're trying to figure out where the bugs are lurking.

[sara]

mark is smart. We also go to a place with no cell reception on our vacations, for this exact same reason. *GRIN* Although we usually figure we can get away with doing that for two nights a year, not two weeks....

[prana]

Thank you for keeping us updated. It is always appreciated.

- arturia (don't wanna risk logging out of the RP account).

[cypher]

Good luck to Fu on the bug hunt, and thank you all for your hard work and vigilance! <3

[marseilles]

Thank you for the update.

[luscious_purple]

I would like to show my thanks and appreciation as well. I wish more of my LJ friends would mirror themselves on DW, so that I wouldn't miss them so much....

[sweetmeow]

I think you may find that more will be cross posting after this last DDoS go-round. A bunch of my LJ friends and I created accts. here a while back, but most had not used them - until now. I had no intention of leaving LJ - and still don't. However, Some of us - including me - are about to cross post here.

[ninetydegrees]

Go team DW! Thank you for keeping us updated! I must say that even if the site is slow it's wonderful to be able to reach it at all so I'm very grateful for that. :)

[wytchcroft]

maximum icon LOVE!! :))