Load issues / site search

[alierak]

As denise mentioned yesterday, we've had various ideas about what to do about the site slowdowns. At this point I think it boils down to needing to buy more servers to keep up with all of you, but I am going to try one more thing before we do that -- temporarily disabling site search (or maybe just search indexing, so you might get old search results) so we can use the search server to handle more ordinary webpage requests. If that works, I'll be certain that more webservers are needed.

I believe I've got the memory shortage under control; by removing some unnecessary software from the servers, and forcing the webservers to restart processes more often, we've been able to handle about 50% more simultaneous requests. Meanwhile fu is continuing to look for the cause of the memory leaks, so that we can (1) improve code quality, because no one likes memory leaks, and (2) allow webserver processes to run longer so they are more efficient.

That said, there's another problem that showed up after tuning the webservers: our loadbalancer freaks out whenever all the webservers are busy and it has to queue up requests, and the overall throughput goes down. I see that as possibly being a bug in Perlbal; mark and I will look into it as soon as we can. I am guessing that if we had more webservers, we'd be less likely to run into this problem.

Thanks for your patience, and for all the kind words on denise's post.

Comments

[kirathaune]

Thank you very much for your hard work!