Better late than never

[alierak]

Since mark gave me access to post here, I might as well say something...

You may have seen an interruption in service earlier today. We received a notice from Slicehost about emergency maintenance on a server that became unresponsive (so they were rebooting it). Our global master database was on that server, so most things on Dreamwidth did not work during the outage. The site returned to normal a few minutes later when the server came back up. We're prepared for a longer downtime or larger server failure with a second global database server and offsite backups, but fortunately none of that proved necessary today.

Ideally there wouldn't be a next time, but just in case, matthew and I will also get access to update Twitter (@dreamwidth).

Comments

[ninetydegrees]

Good news. I didn't see anything on twitter despite the numerous 'Sorry, there was a problem' messages but, at some point, I got a longer message telling me to wait five minutes before sending an e-mail. I did and everything was fine again. :)

[susanreads]

I got the message telling me to wait five minutes before sending an email, and when I tried again a few minutes later I got a blank page, and couldn't get back to the message with the email address. Where should people email if that happens again?

[denise]

You usually don't have to -- we have a monitoring script set up to take care of watching things, and it alerts us as soon as something's broken!

Meanwhile,

[onyxlynx]

Icons have disappeared.

Good luck!

Re: Meanwhile,

[alierak]

Yep, more trouble with the same machine. Mark looked at it and updated the Twitter feed. There's going to be another outage in about an hour to switch over to the other database server.

[primwood]

Good work, guys. I did see your tweets!