Load issues / site search

[alierak]

As denise mentioned yesterday, we've had various ideas about what to do about the site slowdowns. At this point I think it boils down to needing to buy more servers to keep up with all of you, but I am going to try one more thing before we do that -- temporarily disabling site search (or maybe just search indexing, so you might get old search results) so we can use the search server to handle more ordinary webpage requests. If that works, I'll be certain that more webservers are needed.

I believe I've got the memory shortage under control; by removing some unnecessary software from the servers, and forcing the webservers to restart processes more often, we've been able to handle about 50% more simultaneous requests. Meanwhile fu is continuing to look for the cause of the memory leaks, so that we can (1) improve code quality, because no one likes memory leaks, and (2) allow webserver processes to run longer so they are more efficient.

That said, there's another problem that showed up after tuning the webservers: our loadbalancer freaks out whenever all the webservers are busy and it has to queue up requests, and the overall throughput goes down. I see that as possibly being a bug in Perlbal; mark and I will look into it as soon as we can. I am guessing that if we had more webservers, we'd be less likely to run into this problem.

Thanks for your patience, and for all the kind words on denise's post.

Comments

[alierak]

Ooh! Thanks, I really thought Perlbal was doing something more lightweight with those. Turns out reproxying was disabled. I've turned that on, so now we're hitting Apache just to get the mogilefs URL for userpics, but I haven't been able to prove that it's caching yet.

[romikchef]

But it's still involves Apache, which gone deprecated on most of high-load projects I know, especially in serving static content. Apache's threads are too heavy, while Nginx's workers are small, and consume very small amount of CPU and RAM to handle requests.
However, I know nothing about connecting Nginx with Perlbal, if it's even possible.

[alierak]

Well, now that it's actually reproxying, and caching the mogilefs reproxy URLs, it's not hitting Apache for every userpic request, just lighttpd.

[romikchef]

Ah, got it!
Well, at any rate DW runs way better than LJ right now.
Hope we all LJ refugees won't make too much trouble :)
Thank you for letting us new home :)

[pauamma]

Thank you for suggesting that userpics performance enhancement!

[romikchef]

However, the numbers still looks the same to me :)
But let's leave userpics alone, a more important thing came to my mind.
While sorting this issue out, I noticed considerable network latency when connecting to DW servers, which is slightly less than 0,2 sec.
And it reminded me one big deal about Nginx vs. Apache.
Apache's processes are blamed for staying for too long with all it's memory consumed while sending HTTP response to client.
So, Nginx's often role is being a proxy between clients and Apache, letting Apache process to output response body in a blazing speed and waste no time serving slow clients. Meanwhile Nginx buffers the response and feed it to client, consuming very little RAM and CPU, as it's the only thing Nginx was designed for.
This was exactly my own case, when I run a web-server on a very old PC, like Celeron 400Mhz. Apache's been choking at traffic peaks and friend of mine suggested to put Nginx in front of it. And it resolved all the problems, worked like a charm until hardware completely passed away :)

I think this slow clients issue is one of the reasons of memory shortage Denise's been talking about.
And with all we Russians who come to DW after recent LJ slowdown, with such a big latency it can be an issue.
I know that Nginx is not the only solution, but one I am [relatively] familiar with.

And thanks for your gift :)

[pauamma]

I'm pretty sure that perlbal handles buffering between fast Apache servers and slow clients as well. (And if it doesn't, someone is bound to correct me.) I have no idea how it compares with nginx in that respect, though.

And you're welcome. :-)

[romikchef]

Well, your own Firefox can correct you :)
I am talking about dynamically generated content this time, not images.
And User-Agent header says it's Apache serving them. That's the point.
However, whatever revese proxy can just resend that header as well.

[pauamma]

Actually, RFC2616 says proxies have to keep the Server header intact:

If the response is being forwarded through a proxy, the proxy application MUST NOT modify the Server response-header. Instead, it SHOULD include a Via field (as described in section 14.45).