Downtime this morning

[mark]

(For some California local definition of 'morning'!)

About 30 minutes ago one of our databases (sb-db03) locked up and stopped serving traffic. This was an active database, so the site quickly stopped when it could no longer serve requests. Alas.

I have failed us over to a backup database and now everything should be working again.

I'm not sure yet what happened to db03, but am currently investigating and will update this post if I come up with a root cause for the problem. Edit: It's back up and doesn't have any visible problems. Disks are fine, data's intact, etc. The graphs and logs show nothing. We'll have to keep an eye on it and see if it manifests further issues.

Sorry for the trouble, please let me know if you still see any problems!

Comments

[tptigger]

Thanks for being on top of it so quickly! And on an early Saturday morning, no less!

[bluiis]

You guys are amazing.

[angrboda]

Thank you for fixing it so quickly. :) Mysterious why it stopped working.

[solmedes]

What they all said. Thanks for being on top of it as always. =D

[darth_eldritch]

I tried to click in this morning, got the error message, and thought, eh, they're on top of it and will have it fixed in a jiffy.

It's a good feeling and thank you for all the hard work and the open communication.

[halialkers]

No problems here. As always the promptness and effectiveness of the response is appreciated.

[kaberett]

You're amazing - thank you so much!

[jdjunkie]

Thanks for taking such prompt action. Really appreciated. :-)

[chimp]

How cool that you guys managed to fix it so quickly!

[onyxlynx]

Ah-hah!

The Twitter thing hadn't been updated and the "is it down" site swore that Dreamwidth was up, so 'twas a puzzlement.

Thanks for letting us know what was up.

[shadowcat]

Thanks for being on top of this and always letting us know when you know. :D

I think the server just decided it needed a nap. :D

[vexed_wench]

Thank for telling us and fixing it so quick.

[madonnya]

Thanks so much for your hard work! You rock.

[subluxate]

Thank you for keeping us updated! It's great to know you guys will always let us know what's happening.

[auronlu]

Thank you!

(And for this Californian, it's morning until the marine layer burns off. Looks like you've still got another hour to go.)

[badly_knitted]

This is why you guys are the best! Thank you.

[zorkian]

Damn that marine layer! I'm learning to fly and we keep trying to go out towards Half Moon Bay, but the fog makes it impossible. (I fly in the mornings.)

Good luck with it burning off, though. It's already quite sunny here in Mountain View.

Edit: This is Mark, logged in to the wrong account.. bah, can't be bothered to switch back. ;;

[anotherslashfan]

Thank you for fixing this so quickly, it was really impressive! Your service is the best.

[morituri_reverie]

Is that what happened? I went on Down or Not and they said the site was up so I thought it was my computer.

[gwendraith]

Thank you.

[james]

Maybe it just needed a nap. I know I sure could use a nap.

[dusk_to_night]

Thank you for your hard work and diligence!

[auronlu]

oh, that sounds lovely!

I'm within earshot of Disneyland fireworks (I say vaguely), so slightly different set of weather options down here.

[acelightning]

Rather curious, isn't it, that LiveJournal had a database problem that expanded into a massive server problem, with most of the functionality of the entire site borked to some degree or other... which they finally got (mostly) working again just before DW developed a problem?

Cosmic rays? Bedbugs? Chinese military hackers?

[denise]

complete coincidence, i assure you :)

[acelightning]

DW is smart enough not to get the government of a large country mad at you :-)

[auronlu]

Total coincidence. The servers just snuck off to wach Eurovision.

[lucce]

Thanks for the update, Mark. We had faith and knew you'd sort it quickly.
As always keep up the good work :)

[susanreads]

I must have tried to log in at just the wrong time yesterday! When Twitter and isup.me didn't mention the problem, I was going "Panic panic is my internet connection borked again? *checks some other sites* No, just my FAVOURITE SITE. Email support, they'll fix it". And you fixed it!

[vera_j]

Many thanks !!!

[pauamma]

I would hope that Dreamwidth servers at least have better taste than that. :-)

[kaberett]

[Late to the party, sorry] Yep, not just you! :-)

[kaberett]

NO WHAT ARE YOU SAYING

EUROVISION IS /IMPORTANT/ DAMN IT

HAVE YOU FORGOTTEN THE GLORY THAT IS PARTY FOR EVERYBODY