mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
Mark Smith ([staff profile] mark) wrote in [site community profile] dw_maintenance2009-05-01 04:10 pm

Unexpected downtime (5 minutes)

I was paged about the site going down. Looking into it, it was caused by our global master database going offline. I initiated a swap on the global slave to promote it to master and the site was back up a few minutes later.

Slicehost sent the following email:

Dear Slicehoster,

At about 20:00 UTC on 01 May 2009, the server hosting your slice became unresponsive to our monitoring system.

Given the repeated issues this server has experienced, we are performing an emergency server swap to replace the hardware. Once the maintenance is complete, all slices will be restarted.

We apologize for the short notice and any problems this may cause you. Unfortunately this is the best course of action to prevent future problems. If you have any questions, please let us know.

We appreciate your understanding and patience as we deal with this issue. Thank you for your support!

Kind Regards,

Slicehost Support

It's kind of unfortunate that we end up with losing a database with no notice, but it's part of the price paid in a hosting environment like we have.

[personal profile] boosette 2009-05-01 08:15 pm (UTC)(link)
Yay transparency ♥

Thanks for letting us know!
kriski: (bigger on the inside)

[personal profile] kriski 2009-05-01 08:28 pm (UTC)(link)
i second that!

and THANK YOU for working all night on the payment issues.
lienne: Several beads painted with smiling faces. (emotion: friendly)

[personal profile] lienne 2009-05-01 08:32 pm (UTC)(link)
Thirded. It is so unbelievably refreshing to have a site like this run by people who are conscientious and openly communicative.
triadruid: Apollo and the Raven, c. 480 BC , Pistoxenus Painter  (Default)

[personal profile] triadruid 2009-05-01 08:59 pm (UTC)(link)
Y'know, there's a lot of economies of (micro)scale here. DW's members are mostly positive about the service, understanding of its growing pains, and seem to be fairly tech/journal-savvy overall.

If this was LJ, I'm certain that the exact same message would have been met with derision, scorn, and abuse.
lienne: Outer space. (emotion: not in touch with reality)

[personal profile] lienne 2009-05-01 09:02 pm (UTC)(link)
Is LJ in the habit of publishing messages from its hosting service to explain outages? Or rough equivalent, since I hypothesize that LJ hosts itself.

[personal profile] allyourbrains 2009-05-01 09:21 pm (UTC)(link)
It did, a very long time ago when the world was still young.
lienne: A rose lying on a laptop keyboard. (object: rose on keyboard)

[personal profile] lienne 2009-05-01 09:24 pm (UTC)(link)
...I can no longer participate intelligently in this discussion. Your freaking awesome screenname is too distracting.
foxfirefey: A guy looking ridiculous by doing a fashionable posing with a mouse, slinging the cord over his shoulders. (geek)

[personal profile] foxfirefey 2009-05-01 11:47 pm (UTC)(link)
Yes they do!

LJ actually has some really good people on its team.
lienne: Outer space. (emotion: not in touch with reality)

[personal profile] lienne 2009-05-01 11:49 pm (UTC)(link)
And the comments to the latest entry on that community appear to be largely civil.
foxfirefey: Fox stealing an egg. (mischief)

[personal profile] foxfirefey 2009-05-01 11:55 pm (UTC)(link)
Mostly, yes. I imagine comments to most DW things like that will really be along the same lines, our glitches have definitely made some people unhappy, too.
xdawnfirex: (Random - Cyborg Thing)

[personal profile] xdawnfirex 2009-05-02 02:49 am (UTC)(link)
And cat macros.
luludi: (dreamwidth: tori@dreamwidth)

[personal profile] luludi 2009-05-02 04:16 am (UTC)(link)
I've been an LJ user since nearly the beginning, and it wasn't always that way. To be fair, as the years went by (and especially in the last few years) there has been a lot of dishonesty, shady behavior and "pay no attention to that man behind the curtain!" sort of behavior going on over there, which is bound to result in a good many unhappy customers.
tixagon: Happy mawr face ^^ (Default)

[personal profile] tixagon 2009-05-01 08:40 pm (UTC)(link)
fourthed. :D
ladyvox: (Default)

[personal profile] ladyvox 2009-05-01 08:19 pm (UTC)(link)
Thanks for the heads up!
yoochun: (Default)

[personal profile] yoochun 2009-05-01 08:28 pm (UTC)(link)
Thanks for letting us know so quickly. ♥
syderia: lotus Syderia (Default)

[personal profile] syderia 2009-05-01 08:35 pm (UTC)(link)
I just wanted to celebrate not only your quick reply, but the fact that you updated the Twitter page almost as soon as the site went down, and telling us why it went down.

Thanks !
smilingslightly: little brown bat perched on her finger (Default)

[personal profile] smilingslightly 2009-05-01 08:42 pm (UTC)(link)
Hear, hear!

[personal profile] ladylivin 2009-05-01 08:50 pm (UTC)(link)
I agree.
blithespirit: (Default)

[personal profile] blithespirit 2009-05-01 11:31 pm (UTC)(link)
Yep, this. <3 the dreamwidth team. :)
olliefree: (Default)

[personal profile] olliefree 2009-05-01 08:42 pm (UTC)(link)
Thanks for the quick response, and the heads-up on Twitter.
farasha: (Default)

[personal profile] farasha 2009-05-01 08:46 pm (UTC)(link)
Wow you guys are fabulous. Definitely worth the business :)
sid: (pink flower 4)

[personal profile] sid 2009-05-01 08:50 pm (UTC)(link)
omg I didn't understand a word of that! *gazes at you admiringly*

kass: dreamwidth logo in Obama style (dreamwidth)

[personal profile] kass 2009-05-01 08:55 pm (UTC)(link)
It was awesome that the site status page pointed me to twitter, and twitter kept me posted. &hearts!
stormy: βͺ ππŽπ“πˆπ‚π„ ❫ 𝑫𝑢 𝑡𝑢𝑻 𝑻𝑨𝑲𝑬 𝑴𝒀 𝑰π‘ͺ𝑢𝑡𝑺 ⊘ (Default)

[personal profile] stormy 2009-05-01 08:56 pm (UTC)(link)
No worries, it was a very brief downtime! Thanks for the heads up, though.
mythicdove: (xena_ancientgreece)

[personal profile] mythicdove 2009-05-01 09:07 pm (UTC)(link)
Thanks for the heads up :)
iincho: all icons: mewtube @ lj (Default)

[personal profile] iincho 2009-05-01 09:07 pm (UTC)(link)
Cheers for the heads up, guys β™₯ Appreciated!
kfk2: Color Swirl <asd> <asd> ' 'ad a'  (Default)

[personal profile] kfk2 2009-05-01 09:13 pm (UTC)(link)
Just glad that the global slave database server was on a different physical box. Did you ask for that or did it just happen?
kfk2: Color Swirl <asd> <asd> ' 'ad a'  (Default)

[personal profile] kfk2 2009-05-01 09:33 pm (UTC)(link)
Yah, I can totally understand how that would be annoying. Especially because they knew the server had some issues and did not warn you. Glad to hear that only a few servers needed to be moved around though, as it means the rest of the site should be OK. :-)
exor674: Computer Science is my girlfriend (Default)

[personal profile] exor674 2009-05-02 05:19 am (UTC)(link)
Hahaha "Theory"
starwinds: Impertinent-looking Dr. House labeled "I AM MADE OF AWESOME." (Default)

[personal profile] starwinds 2009-05-01 09:42 pm (UTC)(link)
Fabulous response time. Thanks for letting us know about it and resolving it so quickly.
fascination: (Amelie grin.)

[personal profile] fascination 2009-05-01 09:55 pm (UTC)(link)
I missed the downtime, but wow, you're really quite responsive when things *do* go down. :) Big improvement over LJ.
brightwanderer: Guardian Sol from Celestial Chronicle (Default)

[personal profile] brightwanderer 2009-05-01 10:13 pm (UTC)(link)
Quick question (and hello! I am happy to be here!) - is there a system in place for reporting non-urgent bugs/tweaks? There's a bit of formatting on the profile page that probably needs looking at, very much in a "when someone has time" way, but I'm not sure how to check if it's already been noted or where to say something about it given that it's low priority.
hermitty: (Default)

[personal profile] hermitty 2009-05-02 01:47 am (UTC)(link)
I would suggest posting it to the last [site community profile] dw_news post.
afuna: Cat under a blanket. Text: "Cats are just little people with Fur and Fangs" (Default)

[personal profile] afuna 2009-05-02 01:59 am (UTC)(link)
Sure, file a support request (, and someone can sort it out.

cheers! :)
mimbulusmimbletonia: Ryan Rossy (Default)

[personal profile] mimbulusmimbletonia 2009-05-01 11:48 pm (UTC)(link)
I don't mind paying this price.
rroselavy: (Default)

[personal profile] rroselavy 2009-05-02 12:10 am (UTC)(link)
**looks sheepish**

Here I thought I broke the site because I was building a custom journal treatment right in the middle of Dreamwidth going offline.
blair: (Default)

[personal profile] blair 2009-05-02 02:25 am (UTC)(link)
You know you love a place when an outage happens and you think "OH NO is everyone okay!?!?" instead of "HOW DARE THEY."
xdawnfirex: (Random - Kanji)

[personal profile] xdawnfirex 2009-05-02 02:53 am (UTC)(link)
I gotta say, I've been thinking about this most of the day, and it's really kind of an odd thing.

Yesterday when I was trying so hard to buy my account and couldn't because of a bug, I was furious, and I reacted in much the same way as I would have reacted to a similar problem with LJ. Then everyone I dealt with, especially [staff profile] denise, was so nice, that I found myself almost confused.

I'm not accustomed to being treated by my journal hosting service as though I am a customer who matters, and it's almost bewildering.

Thank you for making me go "bzuh?" :)
winged: Final Fantasy X character Yuna, smiling (Default)

[personal profile] winged 2009-05-02 03:56 am (UTC)(link)
Seriously. I posted a couple of bugs that were preventing me from buying my account, and less than ten minutes later each one was fixed. Of course, they were major bugs so I wasn't the only one reporting them, but whoa. I was impressed. And I'm impressed by the transparency of this post too. It's much easier for me to feel forgiving and involved when I actually know what's going on.
kayjayuu: overhead shot of espresso in silver lined cup and saucer (Default)

[personal profile] kayjayuu 2009-05-02 04:12 pm (UTC)(link)
Stupid question, if you don't mind.

You have redundant servers and backups, right? I mean, if the database is ever lost on one server, there are others? And they are in separate physical locations?
ride_4ever: (FK oh noes)

[personal profile] ride_4ever 2017-05-07 02:05 am (UTC)(link)
More unexpected downtime? Attempts to read DW journals and attempts to leave DW support request all lead to a message that says "gateway time-out".