Load issues / site search

[alierak]

As denise mentioned yesterday, we've had various ideas about what to do about the site slowdowns. At this point I think it boils down to needing to buy more servers to keep up with all of you, but I am going to try one more thing before we do that -- temporarily disabling site search (or maybe just search indexing, so you might get old search results) so we can use the search server to handle more ordinary webpage requests. If that works, I'll be certain that more webservers are needed.

I believe I've got the memory shortage under control; by removing some unnecessary software from the servers, and forcing the webservers to restart processes more often, we've been able to handle about 50% more simultaneous requests. Meanwhile fu is continuing to look for the cause of the memory leaks, so that we can (1) improve code quality, because no one likes memory leaks, and (2) allow webserver processes to run longer so they are more efficient.

That said, there's another problem that showed up after tuning the webservers: our loadbalancer freaks out whenever all the webservers are busy and it has to queue up requests, and the overall throughput goes down. I see that as possibly being a bug in Perlbal; mark and I will look into it as soon as we can. I am guessing that if we had more webservers, we'd be less likely to run into this problem.

Thanks for your patience, and for all the kind words on denise's post.

Comments

[archangelbeth]

Thanking you for your work in person, too!

I'm having some crossposting failure issues just now -- do you know if it's part of y'all needing to scale back stuff, or is this a Livejournal interfacing problem? (It says, "Failed to connect to http://www.livejournal.com/interface/xmlrpc.") ...Please excuse me asking in public. I'm kind of figuring it's not just me, and someone will say, "Oh, yeah, that's clearly X" and hopefully you won't get a bazillion other people asking once they see that it's not just them, either. >_>

Again, thank you for all you've managed to do with this!

[alierak]

Yeah, that's probably due to LJ disabling or rate-limiting the feature.

[gusl]

This is happening to me right now!

[glittertine]

Thank you for keeping us up to date! It's really appreciated, even if I'm one of the lucky people who never even noticed anything bad was going on at DW in the first place.

[technoshaman]

Haven't seen much in the way of a slowdown either...

And it bears repeating: The telling thing about DW is that when ever there's a post in News or Maintenance or wherever official, the comment stream is full of "awesome" and generally supportive remarks. Because, well, y'all rock.

[ficwriter1966]

I have no idea what most of that means! :)

But it sounds like you guys are really on top of things, so...thank you! The positive attitude and responsiveness are very much appreciated (if I dare speak on behalf of other members).

[kirathaune]

Thank you very much for your hard work!

[jadedmusings]

Thank you for both the hard work and an effort to keep your users informed of what's happening, even if half of it is like some bizarre alien language to me. ;)

[ariestess]

Thanks for all the work you're doing. And I'll send more virtual cookies, ice cream, and/or alcohol...

[evilawyer]

This is all gibberish to me. I do, however, appreciate that you and the rest of the team are trying to keep me and the rest of the Dreamwidth users apprised of developments. I just wanted to let you know, as this sounds like it's a problem that wouldn't have happened (or at least happened so badly and noticeably) but for the extra load on DW from LJ users whose impatience keeps them from being able to wait until LJ fixes its problems and comes back up to full speed: I appreciate Dreamwidth all the more for (1) being here and (2) having sufficient patience and understanding to put up with and try to accommodate all the LJ regulars who look on DW as the red-headed step-child to be thought of only when LJ isn't meeting their "give me comments now" expectations. As someone who still thinks of the old days of LJ fondly, I really appreciate this.

[gwendraith]

Thank you for the update and your hard work. It does seem much better today, actually.

[primwood]

Thanks, alierak! It's good to know what you think is causing the problems.

[ninetydegrees]

Thanks again for your hard work alierak and for this update (and ironically I wouldn't know there were still some issues without this post because everything has worked well for me for at least two days). I hope you got some rest or will soon.

[wytchcroft]

y'all rule - all your efforts are appreciated :)

[creature_girl08]

As someone who is new here thanks for all your hard work. It is most appreciated. I like seeing a team who makes sure to let people know what is going on. High five!

[mariposa_regina]

Thanks for all your hard work!

Newbee question

[creature_girl08]

Since I am not sure how to ask about this I'll be a newbee and just ask here?

I want to make capitalizations for things such as my entry titles and tags but when they show up everything is in lower case. Help?

Most appreciate someone leading me in the right direction to fix this.

DW rocks,

Re: Newbee question

[denise]

The tags you use on entries are automatically lowercased, to avoid the chance of getting multiple tags when you forget how you capitalized things ("my life", "My Life", "My life", etc).

Entry titles being lowercased is because of the theme you're using on your journal -- some themes play around with text because the designer thought it looked best that way. If you don't want to pick a different theme, but you do want your entry titles to be capitalized the way you want them to be, go to:

http://www.dreamwidth.org/customize/options?group=customcss

and cut and paste this in the big "Use embedded CSS" box:

h3.entry-title {
text-transform: none;
}

That should fix it (I'm doing this from memory and haven't tested it) -- if it doesn't, you can post to style_system asking people how to do it; there are a lot of people who hang out in that community who are much better at that sort of thing than I am :)

[majoline]

Thank you for the up-to-date information! ♥

[geminianeyes]

Thank you for keeping us informed!

[ironed_orchid]

Thanks for keeping us updated.

[impromptu]

Thank you for the update, for your hard work in general, for giving out invites for free accounts...just for the nice atmosphere!))

[musyc]

Thank you for the hard work you've put in to get things working up to speed for us all. Site has been back to its regular quickness for me all day, and your efforts are highly appreciated.

[romikchef]

Just curious, have you ever considered possibility of using NGINX web-server to serve static content? As far as I can see, at least userpics being served from the same host by hulky and memory-hungry Apache. So, it can be issue too.

[pauamma]

I think there's a trick that lets perlbal (the load balancer alierak mentioned in the entry) retrieve userpic data directly from MogileFS (the blob data storage thingy Dreamwidth and LiveJournal use for storing userpics, S2 layouts, and a bunch of other stuff) without involving Apache.</unclear on the details>

[blnchflr]

Best of luck figuring out working solutions, and thanks for keeping us up to speed on what goes on!

[ratcreature]

Good luck with fixing this. I hope search will come back quickly, I use it to find stuff in my journal all the time.

[ithilwen]

Thanks for all your hard work! And thanks for letting us all know in a timely fashion what's causing the problems.