Dec. 30th, 2011

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
Hi all,

I'm doing some updates on our backend file storage system. This should be transparent, but there may be a brief window where icons aren't loading. Please let me know if you are having any trouble or see any problems!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
Hello, my lovely Dreamwidthians! I bring to you information about several site issues this afternoon/evening/other-time-of-day-as-appropriate-for-your-timezone, and the steps we're taking to fix them:

* Some people have noticed intermittent site slowdowns today. This is partially because of increased traffic, and partially, as we feared, because the import queue, and the speed at which we're processing import jobs, is putting a lot of load on the database servers. Short-term fix: [staff profile] mark is juggling things around to reduce database load and trying to direct some queries to the backup database machine. Long-term fix: we have a ticket in with our hosting provider to upgrade the database machines to make them bigger, better, faster, and more.

* You may get some "internal server error" messages when submitting data to the site, or messages saying that the site sent no data or the connection was reset. This is almost certainly related to the new load balancing solution we put into place last night, and Mark is doing some configuration tweaks and fixes that will hopefully stop it from happening. We're not 100% sure on the exact cause -- there are a few things it could be -- but we're going to keep whacking at it until the errors go away.

As part of these fixes, Mark may need to take the site down for a brief maintenance window -- he's hoping he'll be able to make the fixes without having to put the site into maintenance mode, but he might not be able to. The site may be slow for a while in the next few hours, and there may be brief periods of downtime. We're working to fix the problems as fast as we can!

(Well, okay, Mark is working to fix the problems. I'm cheering him on and passing him towels and Gatorade.)
Page generated Sep. 23rd, 2017 07:51 pm
Powered by Dreamwidth Studios