Load issues / site search

[alierak]

As denise mentioned yesterday, we've had various ideas about what to do about the site slowdowns. At this point I think it boils down to needing to buy more servers to keep up with all of you, but I am going to try one more thing before we do that -- temporarily disabling site search (or maybe just search indexing, so you might get old search results) so we can use the search server to handle more ordinary webpage requests. If that works, I'll be certain that more webservers are needed.

I believe I've got the memory shortage under control; by removing some unnecessary software from the servers, and forcing the webservers to restart processes more often, we've been able to handle about 50% more simultaneous requests. Meanwhile fu is continuing to look for the cause of the memory leaks, so that we can (1) improve code quality, because no one likes memory leaks, and (2) allow webserver processes to run longer so they are more efficient.

That said, there's another problem that showed up after tuning the webservers: our loadbalancer freaks out whenever all the webservers are busy and it has to queue up requests, and the overall throughput goes down. I see that as possibly being a bug in Perlbal; mark and I will look into it as soon as we can. I am guessing that if we had more webservers, we'd be less likely to run into this problem.

Thanks for your patience, and for all the kind words on denise's post.

Comments

[archangelbeth]

Thanking you for your work in person, too!

I'm having some crossposting failure issues just now -- do you know if it's part of y'all needing to scale back stuff, or is this a Livejournal interfacing problem? (It says, "Failed to connect to http://www.livejournal.com/interface/xmlrpc.") ...Please excuse me asking in public. I'm kind of figuring it's not just me, and someone will say, "Oh, yeah, that's clearly X" and hopefully you won't get a bazillion other people asking once they see that it's not just them, either. >_>

Again, thank you for all you've managed to do with this!

[alierak]

Yeah, that's probably due to LJ disabling or rate-limiting the feature.

[archangelbeth]

That was my first suspicion. Thank you! (Ha, and now it works. Excellent.)

[andy]

Well, I can say publicly that we at LJ have problems handling long requests (longer than several kilobytes), including posting long entries/comments, uploading pictures, etc., due to DDoS mitigation. Trying to do either of these can end up returning HTTP 500 error from nginx. As usual, our engineers will be working to have these resolved ASAP but I can't give any estimates.

[denise]

You guys have been doing amazing work in dealing with this attack. I said to Carrie and Arie, but I'll say to you, too: if there's anything at all that we can do to help make it easier on you guys, just let us know. This kind of shit is the sort of thing that every site owner should stand up and pitch in on, and anything we can do to help, we will.

Thank you for fighting the good fight :)

[kate_nepveu]

Oh, thanks for the information--I couldn't figure out why one specific post was getting rejected by the crossposter immediately, and that would explain it, it was on the long end of things.

Best wishes on dealing with the DDoS.

[twisted_times]

I'll add another "thank you" for all the LJ staffers doing a thankless task right now. I can only begin to imagine what you guys have had to deal with over the last week or so!

*send tea and buscuits*

[mark]

Just doubling up on what Denise said -- keep up the good fight. If there's anything we can do, don't hesitate to ask. You know how to reach us privately, too, if needed. Good luck.

[havocthecat]

As far as I can see, no one is upset, just confused. Basically, everyone supports LJ doing what they need to in order to stop the DDoS problems happening. (Free speech: A good thing the whole world over.)

[gusl]

This is happening to me right now!

[misskat]

You can open a Support request, but it is probably an issue with LiveJournal, not Dreamwidth. Do open the request though, and I will look at it when I am not on my phone. :)

[denise]

that error message is definitely LiveJournal's problem!