Record breaking traffic day!

[denise]

I'll lead with the good news: we've been setting new records for traffic constantly for the past month or so, and just now we broke 30Mbps (megabits of traffic transfered per second) for the first time, which is an awesome milestone that we are totally dancing around and celebrating. People are using our baby!

So, if we're transfering record amounts of data, and if (as mentioned on our offsite status twitter) we added two new webservers today to help push out traffic, why has the site been so sluggish today?

As I mentioned in the 2 Feb dw-news post, the answer is very complex. There are a lot of underlying causes that can look to you guys (the people who are just trying to load your reading page and comment on various posts) like the site is just plain sucking, and I know it must be tempting to wonder: hey, this keeps happening, why can't they just fix it?



To over-simplify the explanation and use a metaphor: think of the site as the counter at a really busy deli at lunch rush. There's a bunch of people behind the counter taking customer orders, making sandwiches, etc, but there's a whole bunch of people waiting in line for their sandwiches, and every person who walks in joins a long line of people waiting. The deli owner can add more people behind the counter making sandwiches (more webservers, which we added today), they can improve the process of taking orders to speed it up by letting you order commonly-requested sandwiches by number (better caching of content that doesn't change often, like images and CSS and JavaScript), and they can rearrange the stuff behind the counter so sandwich makers can work more efficiently and get the sandwiches made faster (code optimization so the webservers complete requests faster) -- but sometimes there's still going to be a line anyway!

Today (the answer changes from day to day, and sometimes from hour to hour), the biggest slowdown comes from entries with large numbers of comments, because the code that generates those pages has a bunch of inefficiencies that don't start showing up until you see a whole lot of people loading those entries and interacting on them -- the code that generates the page takes time to run, and every additional microsecond it takes, it ties up a process on the webserver it's running on. The webservers can only run so many processes at once, so when there's no free webserver processes available, you ask the site for the page and it has to yell "Hang on a second, I'll get to you as soon as I've got a process free!" at you. Even if you aren't viewing or commenting on an entry with lots of comments, somebody else is, and so the servers wind up being busy. (This is like how if you walk into the deli and just want a root beer, you still have to wait in line with all the people who are buying sandwiches.)

We've known this was going to be a problem for a while, but fixing the underlying cause is going to take a lot of work, because there are a bunch of tiny inefficiences that add up to big problems when they're taken all together.

mark has been making a bunch of code fixes that will speed things up in the short term, and allen is working on the more sweeping code changes that will speed things up in the long term. The code changes Mark just made should help a lot, but high traffic periods (evenings, US time) may continue to be sluggish for a few more days.

We've also added more caching of frequently-accessed and unchanging data: CSS files, JavaScript files, images, and icons are all being cached so they load faster, and served from the fast static content frontend, which takes the burden of serving those off the webservers that are working to build pages. Meanwhile, alierak has been working on optimizing our server response so we can squeak every microsecond out of the servers themselves.

So, in short: We're busy! We're all really sorry about the slowdowns at peak traffic times, and we're working really hard to increase capacity and speed up site performance. Thank you all for your patience (you really are the best users a site ownership team could ask for!) and for continuing to use Dreamwidth. This is a very exciting problem to have. :)

Comments

[ilyena_sylph]

I love your extended analogies!

This one is really fantastic, too.

Best of luck to mark and allen and alierak on all of the stuff they're doing!

[rosalynian]

your openness and communication and friendliness is just so bloody amazing, let me tell you that. i'm so glad i came here.

[mark]

I made a fix this evening that should help a lot with the big comment entries. Here's the CPU usage graph from one of our web servers -- purple is bad.



From a tiny change (http://changelog.dreamwidth.org/1138598.html) that really proves the old truism about knowing where to put the period (or whatever it is, I've forgotten).

[denise]

I love it when you talk graphs at me.

(Also, to address your edit: I believe you mean the joke about "chalk mark, $.50; knowing where to put it: $49,999.50")

[dragondancer5150]

This is a very exciting problem to have. :)

Not that it isn't true, but I have to admit it made me laugh. XD

I can't BEGIN to thank you (and Mark...yesterday? day before?) for taking the time to explain things why/how they're running like they are. Sometimes, that's all a user really wants is to be kept informed. I appreciate the really informal, friendly tone too, as opposed to a faceless and formal, very generalized "we're working on it, not going to go into detail, just believe us" approach.

KEEP UP THE GREAT WORK!

[backtothelight]

I am patient. I can wait. You guys do your thing. (: You're awesome.

[moonvoice]

Thanks for keeping us in the loop; and good luck on getting the site to an increased capacity to deal with the increased traffic. :D

Also, huzzah increased traffic!

[em_kellesvig]

You know, one of these days, I'm going to have to throw real money at you all. I really enjoy the site, the service, and the wonderful attitude you have toward keeping everyone up to date. I'm just, you know, terminally broke. But when I do have money, it's yours. Thanks!

[rezykel]

I really enjoy the site, the service, and the wonderful attitude you have toward keeping everyone up to date.

Now those are words of truth!

[formidable]

Thank you for your hard work and keeping us updated! <3

[pebblerocker]

And the Dreamwidth deli makes the best sandwiches! I'll have extra avocado on mine, please.

[jumpuphigh]

Now I want a sandwich with avocado. Mmmmmm

[rezykel]

I used to work in a fast food chain, I greatly enjoy your metaphor!

And cheers for the update and the hard work you're putting into this site. ♥

[chris]

Thumbs up; rock on!

[sara]

think of the site as the counter at a really busy deli at lunch rush. There's a bunch of people behind the counter taking customer orders, making sandwiches, etc, but there's a whole bunch of people waiting in line for their sandwiches, and every person who walks in joins a long line of people waiting.

...I'm having Baltimore deli flashbacks, now, Dee. You'd better shove a corned beef on rye with dressing through the router, hold the angry Jews. ;>

[angle_on_it]

Every day on Dreamwidth is a double-rainbow day thanks to you DW staffers!

+1

[stakes]

seriously

[storyfan]

I haven't noticed any problems at all.

[milady_dragon]

Would this have anything to do with my network page only loading three entries, all of them dated February 1st?

[amphibologies]

I don't know how often you - or the people on your reading page - post, but at the bottom of one of my read pages it says the page will only display the last thousand entries posted within the last fourteen days. So if no-one's been posting much, the ones on the first might be the only ones it can 'see'?

[phagocytosis]

Thanks for explaining how it works!

[andraste]

When you get right down to it, everyone wanting sandwiches all at once is a good problem to have! Much better than the alternative, anyway.

Thank you for keeping us informed of developments.

[februaryfour]

I really love how I learn so much more about the technology underlying the web just by being a customer here.

[psyco_chick32]

Isn't it amazingly educational? I love Denise's ability to break things down; I like to consider myself pretty technologically-savvy, but these things are beyond me.

[shigeharu]

Thank you guys so much. The transparency and your consistent efforts to explain what's wrong in terms that non-techies will still get is fantastic, and it's one of the many reasons I'm really happy to have come over here.

[sisyphus_rolling]

Ditto

[hi_there_aliens]

Thank you for explaining what's going on and how it works!

[heartbetween]

I just. I haven't said anything so far and just admired silently, but can I just say, you people are amazing?

Every time these and the news posts go up I get starry-eyed over this site all over again. *-*

[dhae_knight_1]

Thank you for the information (and for putting it in non-geek speak). When we know what the problem is and that you're trying to fix it, it's so much easier being patient. :-)

[face_kick]

Deities above how is this site so amazing and run by such amazing people?

Keep on rocking, DW, I'll keep giving you my money, love, and support even through the slowness as long as you keep being this awesome, open, and communicative. ♥

[the]

i live for complex problems and solutions being delivered to me through simple metaphors. it makes me feel like i'm in star trek. another exemplary day on the u.s.s. dreamwidth. thank you, captain denise! o7

Sorry, can't resist....

[sepdet]

Aha! So if Mark could just uncouple the Heisenberg compensators, reroute the trilithium crystals through the plasma drive, and reverse the polarity of the neutron flow we'd be all set!

(simple metaphors, yes...but their explanations of technology were about as straightforward as a tesseract after a fender bender with an Escher triangle and a Klein bottle)

[remindmeofthe]

As a former sandwich maker, I adore this metaphor. And am now picturing the lines at my ex-place of work and experiencing deep sympathy for your servers. *g*

Thanks so much for the communication! I really appreciate your efforts to keep us fully informed, even when I don't always have the knowledge to keep up with them. As others have said, the policy of transparency here is wonderful and shows how much you value your userbase. Thank you!