Load issues / site search

[alierak]

As denise mentioned yesterday, we've had various ideas about what to do about the site slowdowns. At this point I think it boils down to needing to buy more servers to keep up with all of you, but I am going to try one more thing before we do that -- temporarily disabling site search (or maybe just search indexing, so you might get old search results) so we can use the search server to handle more ordinary webpage requests. If that works, I'll be certain that more webservers are needed.

I believe I've got the memory shortage under control; by removing some unnecessary software from the servers, and forcing the webservers to restart processes more often, we've been able to handle about 50% more simultaneous requests. Meanwhile fu is continuing to look for the cause of the memory leaks, so that we can (1) improve code quality, because no one likes memory leaks, and (2) allow webserver processes to run longer so they are more efficient.

That said, there's another problem that showed up after tuning the webservers: our loadbalancer freaks out whenever all the webservers are busy and it has to queue up requests, and the overall throughput goes down. I see that as possibly being a bug in Perlbal; mark and I will look into it as soon as we can. I am guessing that if we had more webservers, we'd be less likely to run into this problem.

Thanks for your patience, and for all the kind words on denise's post.

Comments

[gusl]

This is happening to me right now!

[misskat]

You can open a Support request, but it is probably an issue with LiveJournal, not Dreamwidth. Do open the request though, and I will look at it when I am not on my phone. :)

[denise]

that error message is definitely LiveJournal's problem!