alierak (![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
alierak) wrote in ![[site community profile]](https://www.dreamwidth.org/img/comm_staff.png)
dw_maintenance2011-07-30 03:00 pm
alierak) wrote in dw_maintenance2011-07-30 03:00 pm
Load issues / site search
As ![[staff profile]](https://www.dreamwidth.org/img/silk/identity/user_staff.png)
denise mentioned yesterday, we've had various ideas about what to do about the site slowdowns. At this point I think it boils down to needing to buy more servers to keep up with all of you, but I am going to try one more thing before we do that -- temporarily disabling site search (or maybe just search indexing, so you might get old search results) so we can use the search server to handle more ordinary webpage requests. If that works, I'll be certain that more webservers are needed.
I believe I've got the memory shortage under control; by removing some unnecessary software from the servers, and forcing the webservers to restart processes more often, we've been able to handle about 50% more simultaneous requests. Meanwhilefu is continuing to look for the cause of the memory leaks, so that we can (1) improve code quality, because no one likes memory leaks, and (2) allow webserver processes to run longer so they are more efficient.
That said, there's another problem that showed up after tuning the webservers: our loadbalancer freaks out whenever all the webservers are busy and it has to queue up requests, and the overall throughput goes down. I see that as possibly being a bug in Perlbal;mark and I will look into it as soon as we can. I am guessing that if we had more webservers, we'd be less likely to run into this problem.
Thanks for your patience, and for all the kind words ondenise's post.
denise mentioned yesterday, we've had various ideas about what to do about the site slowdowns. At this point I think it boils down to needing to buy more servers to keep up with all of you, but I am going to try one more thing before we do that -- temporarily disabling site search (or maybe just search indexing, so you might get old search results) so we can use the search server to handle more ordinary webpage requests. If that works, I'll be certain that more webservers are needed.I believe I've got the memory shortage under control; by removing some unnecessary software from the servers, and forcing the webservers to restart processes more often, we've been able to handle about 50% more simultaneous requests. Meanwhile
fu is continuing to look for the cause of the memory leaks, so that we can (1) improve code quality, because no one likes memory leaks, and (2) allow webserver processes to run longer so they are more efficient.That said, there's another problem that showed up after tuning the webservers: our loadbalancer freaks out whenever all the webservers are busy and it has to queue up requests, and the overall throughput goes down. I see that as possibly being a bug in Perlbal;
mark and I will look into it as soon as we can. I am guessing that if we had more webservers, we'd be less likely to run into this problem.Thanks for your patience, and for all the kind words on
denise's post.
no subject
no subject
And it bears repeating: The telling thing about DW is that when ever there's a post in News or Maintenance or wherever official, the comment stream is full of "awesome" and generally supportive remarks. Because, well, y'all rock.