Site slowdowns and internal server errors

[denise]

Hello, my lovely Dreamwidthians! I bring to you information about several site issues this afternoon/evening/other-time-of-day-as-appropriate-for-your-timezone, and the steps we're taking to fix them:

* Some people have noticed intermittent site slowdowns today. This is partially because of increased traffic, and partially, as we feared, because the import queue, and the speed at which we're processing import jobs, is putting a lot of load on the database servers. Short-term fix: mark is juggling things around to reduce database load and trying to direct some queries to the backup database machine. Long-term fix: we have a ticket in with our hosting provider to upgrade the database machines to make them bigger, better, faster, and more.

* You may get some "internal server error" messages when submitting data to the site, or messages saying that the site sent no data or the connection was reset. This is almost certainly related to the new load balancing solution we put into place last night, and Mark is doing some configuration tweaks and fixes that will hopefully stop it from happening. We're not 100% sure on the exact cause -- there are a few things it could be -- but we're going to keep whacking at it until the errors go away.

As part of these fixes, Mark may need to take the site down for a brief maintenance window -- he's hoping he'll be able to make the fixes without having to put the site into maintenance mode, but he might not be able to. The site may be slow for a while in the next few hours, and there may be brief periods of downtime. We're working to fix the problems as fast as we can!

(Well, okay, Mark is working to fix the problems. I'm cheering him on and passing him towels and Gatorade.)

Comments

[denise]

The stats page is public! You can see the old stats or the new stats (which break things down a little differently), or the raw stats from the old stats page (which doesn't separate out new accounts by type-of-account, so it's got communities, OpenID accounts, feed accounts, etc, included in the "newbyday" numbers).

[soc_puppet]

Ooo, the new stats page is super nifty! I'll have to update my bookmarks.

[denise]

*nod* The new stats page was a project that came out of our participation in Google Summer of Code last summer, and it's pretty awesome. There are still some stats that are only available on the old page, but we'll be migrating them all over time, and then we'll be able to make the new stats page into the only stats page!

[ironed_orchid]

OH WOW!

I last looked at the raw stats on the 21nd which was when there was a big jump in the new by day numbers.

I didn't realise that was just the start.

WOW!