site slowness: it's complicated!

[denise]

The site slowness for the past day or so has been due to a bug somewhere in our code that's causing our webserver processes to run out of memory too quickly and lock up the machine.

alierak has been staying on top of things and tweaking the webserver settings to keep things running and to make sure that the settings we're using have the best chance of not running into the "run out of memory, lock up machine" problem. Unfortunately, this means that -- in order to minimize the chance that the site is down entirely -- we've had to seriously lower the number of webserver processes that are running at any time and lower the amount of time before they restart by themselves (and free up the locked-up memory). This means that there are fewer webserver processes available to accept your requests and serve you pages from the site.

Basically, at this point it's a case of "down because of the problem or slow because of the steps we're taking to fix the problem"!

Since it's obvious at this point that just webserver tweaks isn't going to cut it for now, we're doing two things to get the site back to its usual zippy self:

a) Trying to find the root cause of the bug that's making our webserver processes freak out. Memory leaks are really hard to find and debug, which is why it's taking so long. We have a few ideas on how to find what's causing it, and fu is concentrating on that end.

b) Seeing what we can do to get more resources into the webserver pool so that even though the webservers are running out of memory quickly and we have to resource-starve them in order to keep them from checking out entirely, we'll still be able to get pages to load quickly without the delay we're experiencing right now. There's an easy way and a hard way for this, too. (And hopefully, the easy way will help enough that we won't have to get to the hard way.)

(This sort of thing always happens when mark is literally unreachable -- he's on vacation for two weeks in remotest Alaska, with no cell phone reception -- but I wanted to specifically give a massive thank you to alierak, our backup sysadmin, who is doing wonders with the problem.)

Comments

[sova_spliu]

Dear Denise, I really appreciate your care about our comfort:)
I like DM and now have a rest from LJ.
It's better to have a small, but smart company,than big & stupid:)))

[ruisseau]

Perhaps an economy size can of bug spray is in order.

http://www.bedbugmundo.com/wp-content/uploads/2010/09/bed-bug-spay.jpg

[chris]

I particularly appreciate the detailed description at a time when you are on minimal typing... even if you are dictating this to someone else who is typing it for you. (No need to reply to this!)

[denise]

Fortunately, my wrist is very much improved, and that's the critical factor in typing! I can mostly work around the shoulder problems.

(Typing with wrist braces on is annoying, though, I will grant you that.)

[falena]

Thanks for the update! Good luck team DW on fixing the problem!

[badly_knitted]

Thanks for the heads-up! Things have been a little slow at times, but slow is infinitely better than the alternative.

Good luck and I hope you guys don't have to work too hard to fix things, you do a brilliant job.

[epershand]

Thanks for the update. Good luck finding a resolution, all!

[fyreharper]

Three cheers for alierak!

(And the rest of you, too! <3)

[siljamus]

Thank you for both the update and the explanation. Go Team DW!

[illariy]

Aww, I was getting worried about the slow load times. Thank you for the timely updates and keeping us informed and many thanks to alierak and fu for their work!

[sothcweden]

Thanks for the explanation. Good luck with the fix-its. Better done right than fast, if you have to choose, imho.

[cluegirl]

If you've answered this already and I've missed it, please just say, or link, and I'll hush, but -- Has anybody else been struck with the complete failure of the crossposting feature to LJ? Is that directly to do with LJ's DDoS attack, or has there been some change of code that's interfering? I ask, because even when LJ's been up, my posts made here on DW have not been getting through, and if they've made a code change on their side that's jossed the whole thing, I can't help wondering if anybody here's on the problem.

[denise]

The earlier posts in dw_maintenance have some information on that! Because LJ is under DDoS and working to mitigate the attack, and because the site isn't always reachable, crossposting is going to fail more often than not. Even if you can reach LJ from your computer, that doesn't mean that DW can reach LJ at that given moment. The best thing to do is to just leave it be once the crosspost fails, and retry in a few days (by editing the post and checking the crosspost box) once the overload on LJ eases up a bit.

[ficwriter1966]

Thank you for the update! I'm new as of today, and am looking forward to exploring and using the site. :)

[denise]

Welcome to Dreamwidth! I hope you enjoy it here. (And I promise you, the site is usually much faster!)

In terms of exploring, one of my favorite things to do when I have some time to kill on the site is check out the Latest Things page. It's like a firehose of stuff that's going on on the site -- you get to read all kinds of interesting stuff there. (It's also pretty common for people to drop by from the Latest page and comment -- "Here from /latest" is a really common phrase in comments. *G*)

There's also Random Journal and Random Community, which will bring you to a random active journal or community on the site, which can help you to find something interesting to read!

All of the above are linked on the Explore menu, along with a few other nifty things. I hope you enjoy your time here :)

[ovalencia]

It wasn't geared towards me but thanks for this! My journal isn't new but I've been using it more and more very recently, so the info is helpful!

[denise]

Yay, I'm glad! Welcome back to DW, in your case. :)

[nightdog_barks]

Hm, response time just sped up for me! Is it A Sign? :-D

[black_cigarette]

I hope so!

Although at the moment, it's possible it's a Sign that LJ is Back.

[denise]

Actually, I believe it is a sign that alierak is awesome. *g*

[keris]

Memory leakages? Aiiie, we hates them, preciousss! And of course they only happen in the system when it's loaded and has lots of users. I'm glad that for this one I'm on the 'user' side of the system (not often I say that, I usually prefer to be knee-deep in code).

And of course it knows that the person who knows about it is away. Deepest Alaska does sound fun, though.

Best wishes finding and fixing them! And do try to get some sleep sometime (that goes for your admins and especially programmers as well).

[denise]

Of course! At this point we can't even tell when the offending code was introduced -- up until this latest wave of high load, the Apache processes were dying off naturally before the leak could chew up all available memory. So it's not like we can say "oh, this is obviously a problem that was introduced with the last code push, let's start there." Never fails, man. Never fails.

And hey, if you ever do decide that you'd like to come be knee-deep in code with us, we're always happy to have new contributors! We'll even give you a hosted developer environment so you don't need to install the code on your own and struggle with Dependency Hell. (Tempt, tempt, tempt ...)

(Seriously, though, we adore having new contributors. Our general-social irc channel is #dreamwidth on irc.freenode.net and the work-specific one is #dreamwidth-dev; the DW-based comms are dw_dev and dw_dev_training. People tend to fall over themselves being helpful to newbies! We have the best contributor team ever. And of course this is not at all to pressure you, just letting you know that the option is available!)

alierak, the relief sysadmin, has been heroically sacrificing his evenings the past two nights running (he has a Day Job to deal with), but fu and I have totally been making sure to get plenty of rest and downtime. Although my sleep schedule runs on around a 30-36 hour day, so I'm not necessarily sleeping when I 'should' be ...

[keris]

Yup, memory bugs can lurk there dormant for ages before waking up and biting you.

(I'm not awake yet, I read that as "hostile developer environment" *g*. Some developers are very hostile, especially before their third cup of coffee in the morning!)

I don't know what you use, Perl is mentioned and Apache. My main area is low level (C++/C and a bit of Perl holding things together, preference for the insides of OS and comms, not very good at UIs (my 'ideal' UI is a command line with several hundred options *g*), although I have (voluntarily even) done HTML+PerlCGI+MySQL stuff as well). Is there a description somewhere of what you need for DW in terms of languages, development, etc.?

(Yes, I'm being tempted...)

[esteven]

Thank you for the timely updates. It is always good to know what is happening. :D Nothing is worse than not knowing...

Thanks to everyone working on the problem.

[janice_lester]

Just wanted to echo the general sentiments of thanks for letting us know what's up, and for all that you do. Also, the general absence of insane goats around here is very refreshing. (Sorry, Frank, you're not that funny. And the face you pull when you can't find someone's journal entry is frightening.) ;-)

♥