mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
Mark Smith ([staff profile] mark) wrote in [site community profile] dw_maintenance2011-12-29 05:14 pm

Temporary problems loading the site -- resolved.

Dreamwidth was having issues loading for ~15 minutes. This is now resolved.

The summary of the issue is that when you connect to Dreamwidth, you are actually connecting to a bit of software called Perlbal. This software handles routing your request to one of our web servers (we have a bunch) and it also does some other nifty stuff.

The main problem with it is that it's single-threaded. That means that, on the machines we have that have eight or more CPU cores (most modern stuff!), it can only ever run on one of those. This leaves the machine very underutilized -- i.e., it's mostly idle!

This was never a problem for us because even just one core was enough to handle all of the Dreamwidth traffic. At some point we split it up so that static traffic (images, CSS, etc) goes to a second Perlbal instance, but most of the main web traffic still goes through that primary instance.

Today, we finally hit the threshold where Perlbal was taking 100% of the one core it was on and couldn't go any faster. This caused it to queue up requests -- making the site feel really slow. The backend has plenty of capacity, it's just that the frontend wasn't able to go fast enough to handle the traffic.

The fix was to put a much faster load balancer in front and use it to balance traffic to two different Perlbal instances. Now we have a bit of software called Pound that runs in front. We have always been using Pound, but it was only serving SSL requests. Now it is also serving unencrypted HTTP traffic and is then passing that traffic on to two Perlbal instances. In short, it's a load balancer for our load balancers!

This lets us scale more since Pound is an order of magnitude more efficient than Perlbal. By the time we reach the limits of scalability on Pound, we'll have to legitimately move to bigger hardware. (And actually by the time we get there, we will probably be collocating! Exciting!)
wytchcroft: heavent sent (here)

[personal profile] wytchcroft 2011-12-30 03:54 am (UTC)(link)
"crikey Sarge, look at that!"
"'Ere you better 'ave a drop of brandy, it's xmas!" etc.

767 comments!?!?! on Dec 25th?!?!? not exactly a present i'd ask for, sheesh, i'm glad you made it! :)

well, i don't wanna add to the pile here so...
*waves airily and runs away*

denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2011-12-30 03:58 am (UTC)(link)
It was actually a good excuse to hide from my wife's family. *G*
elialshadowpine: (Default)

[personal profile] elialshadowpine 2011-12-30 02:31 pm (UTC)(link)

Well there's at least an upside. ;)
wytchcroft: heavent sent (tara king)

[personal profile] wytchcroft 2011-12-30 02:36 pm (UTC)(link)
LOL! your secret is safe with me!