site slowness: it's complicated!

[denise]

The site slowness for the past day or so has been due to a bug somewhere in our code that's causing our webserver processes to run out of memory too quickly and lock up the machine.

alierak has been staying on top of things and tweaking the webserver settings to keep things running and to make sure that the settings we're using have the best chance of not running into the "run out of memory, lock up machine" problem. Unfortunately, this means that -- in order to minimize the chance that the site is down entirely -- we've had to seriously lower the number of webserver processes that are running at any time and lower the amount of time before they restart by themselves (and free up the locked-up memory). This means that there are fewer webserver processes available to accept your requests and serve you pages from the site.

Basically, at this point it's a case of "down because of the problem or slow because of the steps we're taking to fix the problem"!

Since it's obvious at this point that just webserver tweaks isn't going to cut it for now, we're doing two things to get the site back to its usual zippy self:

a) Trying to find the root cause of the bug that's making our webserver processes freak out. Memory leaks are really hard to find and debug, which is why it's taking so long. We have a few ideas on how to find what's causing it, and fu is concentrating on that end.

b) Seeing what we can do to get more resources into the webserver pool so that even though the webservers are running out of memory quickly and we have to resource-starve them in order to keep them from checking out entirely, we'll still be able to get pages to load quickly without the delay we're experiencing right now. There's an easy way and a hard way for this, too. (And hopefully, the easy way will help enough that we won't have to get to the hard way.)

(This sort of thing always happens when mark is literally unreachable -- he's on vacation for two weeks in remotest Alaska, with no cell phone reception -- but I wanted to specifically give a massive thank you to alierak, our backup sysadmin, who is doing wonders with the problem.)

Comments

[keris]

Memory leakages? Aiiie, we hates them, preciousss! And of course they only happen in the system when it's loaded and has lots of users. I'm glad that for this one I'm on the 'user' side of the system (not often I say that, I usually prefer to be knee-deep in code).

And of course it knows that the person who knows about it is away. Deepest Alaska does sound fun, though.

Best wishes finding and fixing them! And do try to get some sleep sometime (that goes for your admins and especially programmers as well).

[denise]

Of course! At this point we can't even tell when the offending code was introduced -- up until this latest wave of high load, the Apache processes were dying off naturally before the leak could chew up all available memory. So it's not like we can say "oh, this is obviously a problem that was introduced with the last code push, let's start there." Never fails, man. Never fails.

And hey, if you ever do decide that you'd like to come be knee-deep in code with us, we're always happy to have new contributors! We'll even give you a hosted developer environment so you don't need to install the code on your own and struggle with Dependency Hell. (Tempt, tempt, tempt ...)

(Seriously, though, we adore having new contributors. Our general-social irc channel is #dreamwidth on irc.freenode.net and the work-specific one is #dreamwidth-dev; the DW-based comms are dw_dev and dw_dev_training. People tend to fall over themselves being helpful to newbies! We have the best contributor team ever. And of course this is not at all to pressure you, just letting you know that the option is available!)

alierak, the relief sysadmin, has been heroically sacrificing his evenings the past two nights running (he has a Day Job to deal with), but fu and I have totally been making sure to get plenty of rest and downtime. Although my sleep schedule runs on around a 30-36 hour day, so I'm not necessarily sleeping when I 'should' be ...

[keris]

Yup, memory bugs can lurk there dormant for ages before waking up and biting you.

(I'm not awake yet, I read that as "hostile developer environment" *g*. Some developers are very hostile, especially before their third cup of coffee in the morning!)

I don't know what you use, Perl is mentioned and Apache. My main area is low level (C++/C and a bit of Perl holding things together, preference for the insides of OS and comms, not very good at UIs (my 'ideal' UI is a command line with several hundred options *g*), although I have (voluntarily even) done HTML+PerlCGI+MySQL stuff as well). Is there a description somewhere of what you need for DW in terms of languages, development, etc.?

(Yes, I'm being tempted...)

[denise]

*tempt tempt tempt tempt tempt*

DW itself is coded in Perl (the code is at our Mercurial repo), with a MySQL backend. Frontend-wise, obviously there's the HTML and CSS and JS, and we're just moving over now to using Template Toolkit as our templating system (formerly it was BML, which is an unholy homebrew system that Brad Fitzpatrick rolled in 1996 for his first major project and moved along to LJ when he started it). If your Perl is rusty, we've got plenty of people who are willing to tutor. (I am living proof that our mentoring system works; when we started DW, I said that I was just going to be "the suit", and then as we were working to move from closed beta to open beta and trying to knock out as many bugs as possible, I said, hey, if you show me what to do, I can do a few little things, right? Next thing you know, I'm something like #4 on the list of "most patches contributed", and while I'm nowhere near the point of being able to code major features from scratch, I can certainly work on little things!)

Our Bugzilla instance has the list of open "things to be worked on", which can give you a good idea of the kind of things we have "on tap" for people to pick from. There's everything from "change this from a checkbox to a dropdown" type bugs all the way up to "add this new major feature" type bugs, and there's plenty of things both backend and frontend to dig your teeth into.

[keris]

My Perl isn't so much rusty as idiosyncratic *g*. I just looked at the Bugzilla instance...

[denise]

Hooray! If you've got any questions, dw_dev_training is full of people who are happy to help :)

[pauamma]

I think I remember you from somewhere, for some reason. You ever hung out in a.c, a.s.r, or #c?

[keris]

The Scary Devil Monastery? A little, many years ago. Very briefly in alt.callahans (most recently in 2004 via a crosspost from several other newsgroups), I found the volume of traffic there overwhelming. And several times intermittently on alt.comp*, comp.std.c and comp.std.c++ and the like (and in all sort of odd places in alt.* and rec.* and uk.*, particularly the filk and SF fandom related ones). Almost always with keris or keristor somewhere in the email address (and if anyone is using one of those the odds are good that it's me; on LJ keris is someone else, and .org I think is a German medical organisation).

Mind you, in the heyday of Usenet I didn't do Perl, I was C/C++ only (well, plus various assemblers and odd bits of scripting in bash and awk).