Downtime

[mark]

Welcome back.. many of you noticed that we were just down for about an hour.

Anyway, it looks like Perlbal on our main load balancer crashed. Unfortunately, fu didn't have my phone number and wasn't able to raise me. I happened to check my email and saw the problem going on, so hopped online.

I'm really sorry for the outage and how long it took us to get the site back up. That is definitely not the level of service we strive for here.

Going forward: The phone issue has been remedied and I will do another round of broadcasting my contact details to make sure everybody knows how to call me. Also, this raises the question as to why Nagios didn't page me... so I will investigate that and figure out what happened and fix it.

What should have happened is that, within several minutes of the site being down, an automated alert should have fired and paged me. Then I would have hopped online and been able to fix things and we would have only been down for five minutes. That obviously didn't happen, and again I apologize, and will spend some time remedying the backend alerts.

Thanks for your patience. Let me know if anything is broken or otherwise misbehaving.

Comments

[shinsetsu]

Thanks for fixing things as soon as you did. :)

[fu]

Thank you mark! (So far everything looks good now that we're back up)

[omnigrowth]

I think the main thing here is that once you saw what was going on, you fixed the issue as soon as possible. You all still delivered the level of service the best you could under the circumstances, and I personally appreciate that.

Thank you for keeping us updated and being honest. It makes me very happy to be a member of Dreamwidth when I see you guys doing the work that you do to keep the site running, downtime or not. :)

[snottygrrl]

what they said.

seriously, things happen and you did the best you possibly could have with the given challenges and are taking steps to make sure it's new challenges that cause issues in the future.

good on you!

[glittertine]

<3 Thanks for explaining!

[cass404]

Thank you!

And, wow, refreshing to read, "Hey, the system broke and we're sorry. We'll try to learn from it and be better in the future." I honestly admire that in ... anyone.

[dorothean]

Ditto!

[ranunculus]

Honest explainations such as yours go a VERY long way towards making me a happy customer. Settles down purring.

[jb_slasher]

I didn't have time to notice the downtime but I'm impressed that you fixed it as quickly as you could and took the time to inform us, too. Thank you so much for keeping the site Alive and Awesome! :)

[inoru_no_hoshi]

Nagios passed up a chance to page you? I'm...kinda worried. Let me know if I need to make chicken soup for an ill bot? :PPP

More seriously, I didn't notice the downtime, but I'm glad it was taken care of as soon as you noticed!

[ex_moondancer561]

Oh, this is simply inexcusable...you didn't get the page AND still managed to fix the problem. What is this world coming to?

Honestly, you had the site back up before I noticed it was down, and if you hadn't, well, I have a new granddaughter I can look at.

[darkicedragon]

Thank you for keeping us informed!

[aislyn]

*thumbs up* ^_^

[rebeccation]

This sort of response, plus the dw twitter account, makes up for almost any downtime. Thanks!

[andraste]

I am kind of stunned at the idea of Nagios not paging someone. Here at my place of work we seem to spend half the day deleting its messages. (Mainly consisting of 'OMGTHESEVERISBOR ... No, wait, it's not. Carry on.')

[denise]

Our Nagios is characterised as an attention-starved toddler for that very reason :)

[majoline]

Thank you! :)

[ninetydegrees]

I didn't notice any downtime as I was busy sleeping but I don't think I've ever read a better apology. Classy, sir, very classy. *bows*

[samjohnsson]

Not only thank you for getting it fixed as soon as you found out, thank you for letting us know as fast as possible!

[bluiis]

Even if you guys were down an hour, you guys are amazing. You always communicate with us, and you work as quickly as you can to get problems fixed.

[soundingsea]

This kind of attention to service is why I keep telling people DW is the way to go! You guys are great.

(Some days I think we need another nagios instance to watch our nagios... tbd in the Copious Free Time every sysadmin has...)

[gwendraith]

Thanks for letting us know what the problem was. That is good customer care :)

[hobbitbabe]

Thank you for the explanation! I lost some of a long post I was writing, but not all of it, so that's fine.

[falena]

This kind of post makes me so happy to be a paying Dreamwidth customer. There was a problem , you fixed it as soon as possible, you kept us in the loop and you even apologised for any inconvenience caused. This is excellent service, in my book.

[feline_scribe]

There is a reason I very happily passed over some cash for a paid account: this site is amazingly well run and stable. An hour of downtime? Trust me, any time someone can fix something technologically screwed up in under 12 hours, I am happily amazed.

Three cheers for DW, and thanks for all the hard work. :0)

[kyrielle]

I missed the entire hour of down time, being soundly asleep, but I just want to say that I am continuously pleased with how things that mess up are handled at Dreamwidth. You own them, and you commit to making them better, and you keep us informed of what they were. It's so refreshing, compared to so many sites that I use, where you're lucky if they even tell you "hey we're back up" or mention they were down, let alone why or what's being done about it. I REALLY appreciate the transparency.

Also, g'luck figuring out what was up with Nagios.

[ilyena_sylph]

Thanks, Mark!

I was busy sleeping, but I love that your 'whups, it broke' announcements come quick and are pretty complete!

(Bad Nagios, bad!)

[ex_tigre909]

Don't beat yourself up over it, since you still hopped online and fixed it as soon as possible and kept us in the loop.

Thank you so much for your transparency in regards to this issue. It means a lot.