Unexpected downtime (5 minutes)

[mark]

I was paged about the site going down. Looking into it, it was caused by our global master database going offline. I initiated a swap on the global slave to promote it to master and the site was back up a few minutes later.

Slicehost sent the following email:

Dear Slicehoster,

At about 20:00 UTC on 01 May 2009, the server hosting your slice became unresponsive to our monitoring system.

Given the repeated issues this server has experienced, we are performing an emergency server swap to replace the hardware. Once the maintenance is complete, all slices will be restarted.

We apologize for the short notice and any problems this may cause you. Unfortunately this is the best course of action to prevent future problems. If you have any questions, please let us know.

We appreciate your understanding and patience as we deal with this issue. Thank you for your support!

Kind Regards,

Slicehost Support

It's kind of unfortunate that we end up with losing a database with no notice, but it's part of the price paid in a hosting environment like we have.

Comments

[triadruid]

Y'know, there's a lot of economies of (micro)scale here. DW's members are mostly positive about the service, understanding of its growing pains, and seem to be fairly tech/journal-savvy overall.

If this was LJ, I'm certain that the exact same message would have been met with derision, scorn, and abuse.

[lienne]

Is LJ in the habit of publishing messages from its hosting service to explain outages? Or rough equivalent, since I hypothesize that LJ hosts itself.

[allyourbrains]

It did, a very long time ago when the world was still young.

[lienne]

...I can no longer participate intelligently in this discussion. Your freaking awesome screenname is too distracting.

[foxfirefey]

Yes they do!

LJ actually has some really good people on its team.

[lienne]

And the comments to the latest entry on that community appear to be largely civil.

[foxfirefey]

Mostly, yes. I imagine comments to most DW things like that will really be along the same lines, our glitches have definitely made some people unhappy, too.

[tupshin]

http://community.livejournal.com/lj_maintenance/

[rainne]

And cat macros.

[luludi]

I've been an LJ user since nearly the beginning, and it wasn't always that way. To be fair, as the years went by (and especially in the last few years) there has been a lot of dishonesty, shady behavior and "pay no attention to that man behind the curtain!" sort of behavior going on over there, which is bound to result in a good many unhappy customers.