Site outage over

[mark]

Hi all,

The site outage is over. My apologies for the downtime.

One of our databases filled up its disk and went offline, and this caused the site to stop responding. We failed over to the backup database and everything is now back up and running.

Everything should be working. Please let us know if you see any trouble.

We will need to schedule a maintenance window soon to handle the full database and rebuild the cluster so we have a pair again. Stay tuned to this account to watch for announcements about that.

---

Some time last year we realized that our master database pair was filling up its disk, so as part of another downtime we were taking, we cleaned up the slave database and brought it down to around 40% disk usage -- well within comfort.

At the time, we couldn't clean up the master database without taking the site down again or extending the downtime even more, so we decided not to do it at that time and to wait. (Also, it's generally good to separate your maintenances on pairs -- that way if you do something bad and don't notice it, it has time to come out.)

Anyway, the idea was that later we would take another downtime, switch the databases, and then clean up the second machine. That didn't happen though, and the result was that today that database finally ran out of disk space.

Comments

[astro_noms]

Thanks for the awesome response time! <3

[kuwdora]

♥ !

[sharpest_asp]

Dang but you all rock for that response.

[ilyena_sylph]

+1.

I was driving so I didn't even see it.

I got home and there were announcements that DW was down, that you were working on it, and then within seconds of my getting home it was all better!

<3333333333333

[jumble]

Seriously, props on not just the response time but the communication. It is bar none. This is a big part of why I love DW.

[dragondancer5150]

SECONDING

[dil]

Monitoring free space would be a good idea :)

Anyhow thanks for the perfect response time!

[alierak]

Yeah. We also realized we've been graphing but not alerting on disk space :/

[zing_och]

Wow, that was fast! Thank you for letting us know.

[lawless523]

I just tried posting a comment on someone else's journal and wasn't able to.

[denise]

When you say "not able to", what happened?

[hellkitty]

You guys are amazing. Things happen but you were on it immediately *and* communicated with us fast and honestly.

[strega_lyth]

Thanks!

[pretty_panther]

Dayum you guys were quick! Really appreciate your work and keeping us posted!

[solmedes]

Seconding what they all said. That was remarkably quick.

I'm glad it was only disk space, and not something more serious.

[darjeeling]

Not only was the hiccup barely long enough to even register as downtime, you swarmed all over fixing it like a Zergling rush AND we even got technical follow up on what caused it.

You guys are too awesome for words. ♥

[mn]

Thank you very much indeed for fixing this so quickly!

[darth_eldritch]

Thank you!

I knew everything was going to be fine soon!

[justhuman]

Thanks for the quick response and even more for the update. We love that you folks talk to us.

[darth_eldritch]

I can't get on my journal page, except through links to specific entries in my inbox or through tags.

I get the maintainence page when I click on my username, or any link that goes directly to my journal, such as from my favorites bar.

[denise]

Can you try clearing your browser cache? Most likely, your browser is holding onto a cached copy of the page.

[refno619012]

Didn't have any issues whatsoever. :D

[ofearthandstars]

Thanks so much for the clear and quick response! ♥!

[author_by_night]

Still unable to access...

[denise]

Try clearing your browser cache and trying again!

[the]

thanks dudes! tho when i try to go to rpanons (not the reading page or profile, the main page) i still get "We're sorry, Dreamwidth Studios is currently undergoing maintenance or downtime..."

[the]

aaand now it's gone!

[peoppenheimer]

Mad props for communication!! (1) Notice went up on Twitter right away. (2) The problem was diagnosed and fixed very quickly. (3) You gave us a detailed and comprehensible explanation, with a plan for a longer-term fix. You rock!

[danno_williams]

Forgive the RP journal.

I love this place. I'm always willing to give organizations and people more leeway if they just have the guts to be forthcoming about problems and what they are doing to fix it.

So mad props and awesome customer service and all that.

[sara]

You're so courteous to go down while I was having a nap anyway....;>

[cybel]

Since this morning every time I load a new page at amplificathon.dreamwidth.org a generic file downloads to my Mac. It's just called player, no file type indicated. In Get Info it's just called a document, with a default to open in Text Edit. Needless to say I keep trashing the file without opening.

Any idea what might be causing this?

[denise]

Looks like it's your computer or browser trying to download the player for one of the embedded files, probably because of an extension or something?

I didn't even notice.

[babysprite]

Honestly, I didn't notice.

By the way, Dreamwidth looks and works great with the w3m web browser.

Yours from the ba$h shell,

/\__/\
(='.'=)
(")_(")

babysprite

Re: I didn't even notice.

[yvi]

Oh, someone using a text-browser, awesome!

(I once tried my luck with Livejournal and Lynx when I broke my XServer - I don't remember how well that worked :))