Unexpected downtime (5 minutes)

[mark]

I was paged about the site going down. Looking into it, it was caused by our global master database going offline. I initiated a swap on the global slave to promote it to master and the site was back up a few minutes later.

Slicehost sent the following email:

Dear Slicehoster,

At about 20:00 UTC on 01 May 2009, the server hosting your slice became unresponsive to our monitoring system.

Given the repeated issues this server has experienced, we are performing an emergency server swap to replace the hardware. Once the maintenance is complete, all slices will be restarted.

We apologize for the short notice and any problems this may cause you. Unfortunately this is the best course of action to prevent future problems. If you have any questions, please let us know.

We appreciate your understanding and patience as we deal with this issue. Thank you for your support!

Kind Regards,

Slicehost Support

It's kind of unfortunate that we end up with losing a database with no notice, but it's part of the price paid in a hosting environment like we have.

Comments

[boosette]

Yay transparency ♥

Thanks for letting us know!

[kriski]

i second that!

and THANK YOU for working all night on the payment issues.

[lienne]

Thirded. It is so unbelievably refreshing to have a site like this run by people who are conscientious and openly communicative.

[triadruid]

Y'know, there's a lot of economies of (micro)scale here. DW's members are mostly positive about the service, understanding of its growing pains, and seem to be fairly tech/journal-savvy overall.

If this was LJ, I'm certain that the exact same message would have been met with derision, scorn, and abuse.

[lienne]

Is LJ in the habit of publishing messages from its hosting service to explain outages? Or rough equivalent, since I hypothesize that LJ hosts itself.

[allyourbrains]

It did, a very long time ago when the world was still young.

[lienne]

...I can no longer participate intelligently in this discussion. Your freaking awesome screenname is too distracting.

[foxfirefey]

Yes they do!

LJ actually has some really good people on its team.

[lienne]

And the comments to the latest entry on that community appear to be largely civil.

[foxfirefey]

Mostly, yes. I imagine comments to most DW things like that will really be along the same lines, our glitches have definitely made some people unhappy, too.

[tupshin]

http://community.livejournal.com/lj_maintenance/

[rainne]

And cat macros.

[luludi]

I've been an LJ user since nearly the beginning, and it wasn't always that way. To be fair, as the years went by (and especially in the last few years) there has been a lot of dishonesty, shady behavior and "pay no attention to that man behind the curtain!" sort of behavior going on over there, which is bound to result in a good many unhappy customers.

[tixagon]

fourthed. :D

[cookie]

Didn't even see it, heh.

[ladyvox]

Thanks for the heads up!

[yoochun]

Thanks for letting us know so quickly. ♥

[syderia]

I just wanted to celebrate not only your quick reply, but the fact that you updated the Twitter page almost as soon as the site went down, and telling us why it went down.

Thanks !

[smilingslightly]

Hear, hear!

[ladylivin]

I agree.

[blithespirit]

Yep, this. <3 the dreamwidth team. :)

[olliefree]

Thanks for the quick response, and the heads-up on Twitter.

[farasha]

Wow you guys are fabulous. Definitely worth the business :)

[sid]

omg I didn't understand a word of that! *gazes at you admiringly*

:-)

[kass]

It was awesome that the site status page pointed me to twitter, and twitter kept me posted. &hearts!

[stormy]

No worries, it was a very brief downtime! Thanks for the heads up, though.

[mythicdove]

Thanks for the heads up :)

[iincho]

Cheers for the heads up, guys ♥ Appreciated!

[kfk2]

Just glad that the global slave database server was on a different physical box. Did you ask for that or did it just happen?

[mark]

In theory everything is supposed to be on separate boxes. This incident has revealed that it doesn't seem to be a guarantee. Since then though I've emailed back and forth with Slicehost support a few times, and they gave me a list of what is sharing hardware now, and offered to break them up so as not to share hardware.

There are three slices being moved right now to get them on separate hardware and then everything will be on a unique host...

But yeah. Annoying!

[kfk2]

Yah, I can totally understand how that would be annoying. Especially because they knew the server had some issues and did not warn you. Glad to hear that only a few servers needed to be moved around though, as it means the rest of the site should be OK. :-)

[exor674]

Hahaha "Theory"

[starwinds]

Fabulous response time. Thanks for letting us know about it and resolving it so quickly.

[milkymoon]

I missed the downtime, but wow, you're really quite responsive when things *do* go down. :) Big improvement over LJ.

[brightwanderer]

Quick question (and hello! I am happy to be here!) - is there a system in place for reporting non-urgent bugs/tweaks? There's a bit of formatting on the profile page that probably needs looking at, very much in a "when someone has time" way, but I'm not sure how to check if it's already been noted or where to say something about it given that it's low priority.

[hermitty]

I would suggest posting it to the last dw_news post.

[afuna]

Sure, file a support request (http://www.dreamwidth.org/support/submit.bml), and someone can sort it out.

cheers! :)

[nev_ertheless3]

I don't mind paying this price.

[rroselavy]

**looks sheepish**

Here I thought I broke the site because I was building a custom journal treatment right in the middle of Dreamwidth going offline.

[blair]

You know you love a place when an outage happens and you think "OH NO is everyone okay!?!?" instead of "HOW DARE THEY."

[rainne]

I gotta say, I've been thinking about this most of the day, and it's really kind of an odd thing.

Yesterday when I was trying so hard to buy my account and couldn't because of a bug, I was furious, and I reacted in much the same way as I would have reacted to a similar problem with LJ. Then everyone I dealt with, especially denise, was so nice, that I found myself almost confused.

I'm not accustomed to being treated by my journal hosting service as though I am a customer who matters, and it's almost bewildering.

Thank you for making me go "bzuh?" :)

[winged]

Seriously. I posted a couple of bugs that were preventing me from buying my account, and less than ten minutes later each one was fixed. Of course, they were major bugs so I wasn't the only one reporting them, but whoa. I was impressed. And I'm impressed by the transparency of this post too. It's much easier for me to feel forgiving and involved when I actually know what's going on.

[kayjayuu]

Stupid question, if you don't mind.

You have redundant servers and backups, right? I mean, if the database is ever lost on one server, there are others? And they are in separate physical locations?

[ride_4ever]

More unexpected downtime? Attempts to read DW journals and attempts to leave DW support request all lead to a message that says "gateway time-out".