Dreamwidth Maintenance

Site slowness expected shortly

Sun, 29 Apr 2012 00:25:26 GMT

Hi all,

Summary: some unexpected but required maintenance is going to cause the site to slow down by some amount for the next hour or two.

Some details: We have run into an issue with one of our larger communities that requires us to make a change to the way we store comments in memcache. This means that our caches are all going to invalidate here in a few minutes when I roll the code out, so the databases are going to get very busy until we repopulate the caches.

Those of you who have been on LJ for a few years might remember when ONTD ran into an issue when they hit 16,777,215 comments and LJ had to do the same thing. I helped LJ with the code to fix the problem, but for some reason, never committed it on Dreamwidth. Now we have hit the same issue and I feel silly for having not done that, but here we are, and hence this maintenance.

Please let me know if you see any major problems, and again, sorry for the inconvenience. This really isn't a great time to be doing site maintenance.

Update: Congratulations to sixwordstories for being the first community or journal to hit the 16.7 million comment limit in the database. :)

At any rate, currently the community is being migrated to a database where I fixed up the column widths so they won't have this limit anymore. It will take some time to move that much data, though. In the meantime the community will be in read-only mode.

I'm sorry that we hit this -- I really thought we had made this fix on Dreamwidth after LJ ran into it. It turns out we didn't, though, hence this situation. Alas.

comments

database maintenance

Tue, 24 Apr 2012 19:33:23 GMT

Hi all,

Over last night and this morning I've done some database maintenance on our global slave. This should be transparent and cause no problems, but please let me know if you see anything amiss.

For the technically curious, I had to do a dump-and-reload on our MySQL instance so that I could free up space in the InnoDB data file. It had grown to ~240GB back when the machine was also home to the user data, but now that it is just used for global data, it is only about 8GB.

After things are settled for a few days, I will have to perform the same operation on the global master. That will be slightly more exciting, but should cause no downtime either.

comments

Search indexing temporarily paused; search will be down sometime in the next day

Tue, 24 Apr 2012 03:24:27 GMT

We've got a disk that looks like it's about to fail in the machine we use to power site and journal search. We've turned off search indexing temporarily to reduce the number of writes to the disk (which will hopefully let us baby it along for a little while longer), and will be taking search entirely offline for a four-hour maintenance window with our hosting provider as soon as we can schedule a convenient time for the replacement -- either later this evening, or early tomorrow morning.

Indexing being disabled means that no new posts will show up in site search until after the replacement, and when the machine is entirely offline, visiting site or journal search will return a message saying that the feature is temporarily offline. Once the disk is replaced, it may take a day or two for the indexing to catch up with the stuff that was posted while indexing was off; we've been hovering around the point where it takes longer to run the script than the frequency with which we run the script, so it may take us a little while to catch up.

The data on the disk is completely backed up (and in a worst case scenario could be re-indexed anyway), so there won't be any risk. The only thing you should notice will be no new posts appearing in the search results until the replacement, and then a period of downtime while the replacement is performed.

(EDIT, 24 April 5AM EDT: The disk has now been replaced, the data is copied back over, and the indexing is running again and catching up on what was missed. And, for bonus points, mark found a place to optimize a bunch of database queries.)

comments

notifications delay

Thu, 19 Apr 2012 08:00:09 GMT

Thanks to a combination of heavy traffic and a side effect of moving some databases, we had a problem with notifications not being delivered (both via email and the on-site inbox). mark has identified the issue and fixed the underlying problem. The delayed notifications should be sent out now as the workers catch up with the backlog.

We're sorry about the problem!

EDIT, 7AM EDT: after fixing the problem with notifications in general being severely delayed, we're now seeing that AOL has placed us on their "you send us too much mail" blacklist and is refusing all mail from us. We'll see what we can do to get the block listed, but we can't guarantee anything. If you use an @aol.com or @aim.com email address, you may want to temporarily switch your email address to a different email address under your control in order to be sure you don't miss mail.

comments

Code pushed!

Sat, 24 Mar 2012 05:20:50 GMT

The code push is now complete! Please report any problems here for the next 24 hours or so (and to Support after that).

Code tours describing changesets included in this push are:

* 22 Feb - 13 Mar
* 14 Mar - 23 Mar

In particular, we're asking people to turn on the "New JS on Journals" option on the Beta Features page. We've finished converting all the JavaScript inside your journal to JQuery, the more modern JavaScript library we're moving towards. The version available for beta testing now is the release candidate. With the next code push, we will be opting everyone into that beta (with the ability to still opt out of the beta version if you discover a showstopper bug that prevents you from using your journal); with the code push after that, the code will be taken out of beta and you will no longer be able to turn off the new JavaScript behavior at all.

So, turn on the beta now, and report any bugs with JavaScript in your journal to the dw_beta "Last Call" post if you're sure it's related to the new JavaScript, or here if you aren't sure whether or not it's an issue with the new JS!

comments

code push imminent

Sat, 24 Mar 2012 04:51:28 GMT

We are now beginning a code push! We'll update again when we're finished.

comments

Code push tonight

Fri, 23 Mar 2012 15:18:15 GMT

We're planning a code push tonight! We'll begin the process at around 1AM EDT Sat 24 Mar/10PM PDT Fri 23 Mar (5AM GMT Sat 24 Mar).

We'll update both here and in the dreamwidth Twitter offsite status feed when we're ready to begin.

comments

Testing the Fastly CDN

Wed, 07 Mar 2012 23:23:32 GMT

Hi all,

This has now been rolled back. It's not working for some subset of users and that is problematic. We are back to how we were earlier!

I've now rolled out a small change that sends a lot of static traffic through Fastly, an interesting CDN service founded by a guy who I used to work with on LiveJournal. It's a neat service that should give us some really good performance gains -- particularly for users who aren't near our main servers in Dallas.

Edit: If you are using something like NoScript or a similar service that whitelists URLs, you will need to add the following domain:

s.dreamwidth.org.a.prod.fastly.net

Many of you noticed that there was about 15 minutes of broken JavaScript on the site. This showed up as cut tags not expanding, the login form not working, and some other weird behaviors. It's fixed now and was caused by me making a typo in the configuration. Sorry about that.

Please let me know if anything else is broken or not working. Also, if the site seems faster, I'd love to know that too. I'm sure crucially would be interested to hear our opinions of it. :)

comments

code push shortly

Wed, 22 Feb 2012 06:48:57 GMT

We'll be beginning a code push in about 15-20 minutes. ~~Please put up your seat backs and return your tray tables to the full and upright locked position.~~ We'll update this entry when we're done!

(2:40AM EST: As always, the prep turns out to be more involved than we predicted. We'll hopefully be starting soon.)

3:10AM EST: And, we're done! Please report any issues here or to Support.

comments

code push scheduled

Tue, 21 Feb 2012 06:23:57 GMT

We're currently planning to do a code push tomorrow night, tentatively scheduled for 2AM EST Wed 22 Feb/11PM PST Tue 21 Feb (7AM GMT Wed 22 Feb). Most changes scheduled for this push are minor and there shouldn't be any more than a few minutes' disruption to service.

We'll update when we begin for you to report any potential issues you notice.

comments

database rebalancing

Tue, 14 Feb 2012 19:11:19 GMT

A few weeks ago we deployed a new pair of databases. Now that they've broken in and I'm confident in them, I've started the process of rebalancing users to even out the database load. This involves moving users from the old database machines to the new ones.

I don't expect there to be any issues, but as always, please let me know if you see anything untoward!

comments

problems connecting to Dreamwidth

Mon, 13 Feb 2012 22:59:08 GMT

We've seen a handful of people reporting that they're having problems connecting to Dreamwidth. We're trying to narrow down the problem and figure out if it's a problem with the routing on our end or if a major ISP is having routing issues reaching us. (Right now, it's looking like the fault isn't in our network, but there may be steps we can take to fix things if we can identify what's going on.)

If you know someone who's having problems reaching DW, please get them to email support@dreamwidth.org with the results of a traceroute from their computer to dreamwidth.org. That will help us identify where the problems are.

Edit: Strike that! mark found something earlier today that might have explained the problem and fixed it. If you're still getting the problems after this point (6PM EST, 2/13), email us traceroutes then, but most people who were having the problem should notice it's gone away now.

comments

Record breaking traffic day!

Fri, 10 Feb 2012 04:54:59 GMT

I'll lead with the good news: we've been setting new records for traffic constantly for the past month or so, and just now we broke 30Mbps (megabits of traffic transfered per second) for the first time, which is an awesome milestone that we are totally dancing around and celebrating. People are using our baby!

So, if we're transfering record amounts of data, and if (as mentioned on our offsite status twitter) we added two new webservers today to help push out traffic, why has the site been so sluggish today?

As I mentioned in the 2 Feb dw-news post, the answer is very complex. There are a lot of underlying causes that can look to you guys (the people who are just trying to load your reading page and comment on various posts) like the site is just plain sucking, and I know it must be tempting to wonder: hey, this keeps happening, why can't they just fix it?

( in which I make an extended metaphor about site traffic )

mark has been making a bunch of code fixes that will speed things up in the short term, and allen is working on the more sweeping code changes that will speed things up in the long term. The code changes Mark just made should help a lot, but high traffic periods (evenings, US time) may continue to be sluggish for a few more days.

We've also added more caching of frequently-accessed and unchanging data: CSS files, JavaScript files, images, and icons are all being cached so they load faster, and served from the fast static content frontend, which takes the burden of serving those off the webservers that are working to build pages. Meanwhile, alierak has been working on optimizing our server response so we can squeak every microsecond out of the servers themselves.

So, in short: We're busy! We're all really sorry about the slowdowns at peak traffic times, and we're working really hard to increase capacity and speed up site performance. Thank you all for your patience (you really are the best users a site ownership team could ask for!) and for continuing to use Dreamwidth. This is a very exciting problem to have. :)

comments

static file hiccup details

Thu, 09 Feb 2012 03:19:46 GMT

Hi all,

We are currently working around a problem with our static file serving -- i.e., JavaScript, CSS, and image files. I had to move these back to our main web frontend instead of using the fast static frontend.

The root cause is that our hosting provider, ServerBeach, reassigned the IP address we were using for static content. They assigned it to one fo the new machines they are building for us. Given the vagaries of networks, I can't unassign that IP easily on my end. I have to kill the machine and it's still in provisioning, so I don't have access to it yet.

In order to work around this problem, I've had to make two changes -- use the main Perlbal infrastructure and turn off an optimization we use -- this is going to cause things to load a little slower. I will be working this evening to resolve the problem in a more efficient way.

My apologies for the issue here. I don't know why they reassigned our IP address, but as soon as I figure out what happened, I will let you know.

Thanks for your patience.

comments

capacity update

Mon, 06 Feb 2012 23:23:50 GMT

Just to keep you all in the loop -- the servers have been really busy lately! alierak, one of our sysadmins, has been working on things and found an issue with our load balancer. He fixed that, and that has resolved a number of issues people have had connecting to Dreamwidth during periods of peak load.

(For the technical, the issue was that our connection tracking table in the kernel was set to the default 64k, and we've started peaking past that. He raised the limits.)

We have also submitted an order for two more web servers. This will make things faster by just giving us more horsepower to work with. The rest of the system (databases, memcache, network, etc) isn't anywhere near capacity, but we're running low on CPU on our web servers.

We also have some of our developers working on optimizations for entries that have many comments. That work is in progress and we will continue to iterate to make things faster.

If you have any questions, let me know! I'll do my best to answer them quickly and accurately. :)

comments

Code push shortly.

Sun, 22 Jan 2012 23:40:55 GMT

Hi all,

I'm going to be doing a code push shortly. As always, if you see anything stop working or break, please comment here and we'll get it fixed up. Thank you!

Edit: We had an issue with delayed notifications. They should be moving normally again!

comments

Scalability and such

Sun, 01 Jan 2012 00:00:41 GMT

I made a post on my own journal that talks about the scalability/load/capacity of Dreamwidth. The short summary is that we're nowhere near our ultimate capacity on anything and the few things that have started to hit capacity are being expanded. I am very comfortable with our current and future status!

More details here:

http://mark.dreamwidth.org/21787.html

I'm going to disable comments here and ask that if you want to comment/ask questions/etc that you do it over there on the relevant post.

Thank you!

comments

Site slowdowns and internal server errors

Sat, 31 Dec 2011 00:54:13 GMT

Hello, my lovely Dreamwidthians! I bring to you information about several site issues this afternoon/evening/other-time-of-day-as-appropriate-for-your-timezone, and the steps we're taking to fix them:

* Some people have noticed intermittent site slowdowns today. This is partially because of increased traffic, and partially, as we feared, because the import queue, and the speed at which we're processing import jobs, is putting a lot of load on the database servers. Short-term fix: mark is juggling things around to reduce database load and trying to direct some queries to the backup database machine. Long-term fix: we have a ticket in with our hosting provider to upgrade the database machines to make them bigger, better, faster, and more.

* You may get some "internal server error" messages when submitting data to the site, or messages saying that the site sent no data or the connection was reset. This is almost certainly related to the new load balancing solution we put into place last night, and Mark is doing some configuration tweaks and fixes that will hopefully stop it from happening. We're not 100% sure on the exact cause -- there are a few things it could be -- but we're going to keep whacking at it until the errors go away.

As part of these fixes, Mark may need to take the site down for a brief maintenance window -- he's hoping he'll be able to make the fixes without having to put the site into maintenance mode, but he might not be able to. The site may be slow for a while in the next few hours, and there may be brief periods of downtime. We're working to fix the problems as fast as we can!

(Well, okay, Mark is working to fix the problems. I'm cheering him on and passing him towels and Gatorade.)

comments

Updating MogileFS and other backend code

Fri, 30 Dec 2011 19:22:21 GMT

Hi all,

I'm doing some updates on our backend file storage system. This should be transparent, but there may be a brief window where icons aren't loading. Please let me know if you are having any trouble or see any problems!

comments

Temporary problems loading the site -- resolved.

Fri, 30 Dec 2011 01:21:28 GMT

Dreamwidth was having issues loading for ~15 minutes. This is now resolved.

The summary of the issue is that when you connect to Dreamwidth, you are actually connecting to a bit of software called Perlbal. This software handles routing your request to one of our web servers (we have a bunch) and it also does some other nifty stuff.

The main problem with it is that it's single-threaded. That means that, on the machines we have that have eight or more CPU cores (most modern stuff!), it can only ever run on one of those. This leaves the machine very underutilized -- i.e., it's mostly idle!

This was never a problem for us because even just one core was enough to handle all of the Dreamwidth traffic. At some point we split it up so that static traffic (images, CSS, etc) goes to a second Perlbal instance, but most of the main web traffic still goes through that primary instance.

Today, we finally hit the threshold where Perlbal was taking 100% of the one core it was on and couldn't go any faster. This caused it to queue up requests -- making the site feel really slow. The backend has plenty of capacity, it's just that the frontend wasn't able to go fast enough to handle the traffic.

The fix was to put a much faster load balancer in front and use it to balance traffic to two different Perlbal instances. Now we have a bit of software called Pound that runs in front. We have always been using Pound, but it was only serving SSL requests. Now it is also serving unencrypted HTTP traffic and is then passing that traffic on to two Perlbal instances. In short, it's a load balancer for our load balancers!

This lets us scale more since Pound is an order of magnitude more efficient than Perlbal. By the time we reach the limits of scalability on Pound, we'll have to legitimately move to bigger hardware. (And actually by the time we get there, we will probably be collocating! Exciting!)

comments

Code push soon

Thu, 29 Dec 2011 21:46:40 GMT

I'm going to be doing a code push shortly as soon as I'm happy with the positioning of my ducks. (I don't need them to all have the same Y coordinates, but they should at least be near each other.)

This is a smaller push than most, there isn't a ton of stuff that's changed. There are a few major changes to how the importer works though, which is my area of most concern and where I'll be spending a lot of time watching.

As always, please comment and let me know if there are any issues. Thank you!

Update: Code pushed. There was some temporary slowness, our load balancer (Perlbal) got into a weird state where it was using 100% CPU. I restarted it and things returned to normal.

Update #2: There was an issue affecting status messages for community imports. This should be fixed now.

comments

import queue delays

Wed, 21 Dec 2011 23:00:27 GMT

Thanks to a few people reporting problems with their import jobs stuck on the "verify" step for an extended period of time, we discovered a bottleneck in the import process we hadn't realized. We've taken steps to fix it. If your import was showing as "ready to be inserted into the queue", those jobs are now being moved into the import queue more quickly. (That's why, if you've been watching the queue on the Import Journal page, the numbers just jumped like whoa.)

It will take time for the importer to process all the queued jobs -- whenever there's a surge in account creation, there's a corresponding surge in import jobs -- but fear not, once they're scheduled your import jobs will run. You don't have to leave the page open: just schedule the job and wander off, and sooner or later you will look at your journal and all of your stuff will be there like magic. :)

( More of the technical details, for those who are curious )

EDIT, 8:40PM EDT: Sorry about the rampant internal server error problems -- we thought it was a problem with the new webserver, but it turned out that imports were happening too fast and were locking up the database. Mark has throttled back the import speed enough that the errors should go away now. (This means that imports will be happening more slowly, but the queue's backed up enough right now that it probably won't make much difference anyway!)

EDIT, 4:30 PM EDT, 12/23: As always happens whenever we have an influx of new users, the import queue is very, very busy right now. Your import will almost certainly take at least a day to finish. Please be patient! Once your job is in the queue, it will complete eventually and you don't need to stay logged into the site or leave your computer on. Just start it and go do other things, and eventually your stuff will catch up with you. :)

comments

Adding a webserver

Wed, 21 Dec 2011 19:46:03 GMT

Hi all,

Dreamwidth is seeing some extra traffic today, so we're rolling out a new webserver to handle the extra load. We'll be paying close attention to the system load though and will be doing what we can to make sure everything is speedy and working.

Also, today I am joined by my infant son Oliver, who is helping me with the servers. Hello from Mark and Oliver!

comments

Back up!

Tue, 13 Dec 2011 17:52:18 GMT

Looks like we're back online. Please let me know if anything is not working like you expect!

Update: DNS was broken. This affected imports, crossposts, and emails. It's back up and running now and the email backlog will slowly be cleared. Nothing was lost, just delayed.

comments

Extended downtime in 45 minutes

Tue, 13 Dec 2011 16:08:49 GMT

Hi everybody.

The extended downtime I posted about a few days ago is upon is. Dreamwidth will be going offline in about 45 minutes and will be down for about two hours.

Thanks for all of your patience. I'll see you on the flip side.

comments