Addressing the site slowdowns

[denise]

As a few commenters to the previous post noted, we've managed to fix some of the site slowdown issues! alierak has spent most of his night trying to tweak and tune every last bit of performance out of our various servers, and his efforts have really paid off. There still may be some slowdowns -- we're still trying to figure out where the memory leaks in the code are -- but conditions should be considerably improved at this point.

In order to speed up the site, we've turned the machine that handles site search into an additional webserver temporarily. Right now, the two functions -- search database and webserver -- are coexisting side-by-side, but if performance is getting gnarly again, we might need to temporarily disable search in order to commit that machine's resources entirely to webserving.

In the meantime, although search is still enabled, we've had to lower the amount of that server's resources dedicated to the search processes, so you might get a few timeouts or some wonky results here and there if you're trying to use the search function. If that happens, we're really sorry! The situation is temporary, and we'll have the machine back to doing only search as quickly as we can find the root cause of the memory leaks we're dealing with.

Thanks, all, for your patience, and super special thanks with sprinkles on top to alierak for sacrificing his whole night (and then some) to doing battle with the problem.