mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
Mark Smith ([staff profile] mark) wrote in [site community profile] dw_maintenance2013-01-13 02:45 pm

Site outage over

Hi all,

The site outage is over. My apologies for the downtime.

One of our databases filled up its disk and went offline, and this caused the site to stop responding. We failed over to the backup database and everything is now back up and running.

Everything should be working. Please let us know if you see any trouble.

We will need to schedule a maintenance window soon to handle the full database and rebuild the cluster so we have a pair again. Stay tuned to this account to watch for announcements about that.



Some time last year we realized that our master database pair was filling up its disk, so as part of another downtime we were taking, we cleaned up the slave database and brought it down to around 40% disk usage -- well within comfort.

At the time, we couldn't clean up the master database without taking the site down again or extending the downtime even more, so we decided not to do it at that time and to wait. (Also, it's generally good to separate your maintenances on pairs -- that way if you do something bad and don't notice it, it has time to come out.)

Anyway, the idea was that later we would take another downtime, switch the databases, and then clean up the second machine. That didn't happen though, and the result was that today that database finally ran out of disk space.
astro_noms: (cute owl)

[personal profile] astro_noms 2013-01-13 10:53 pm (UTC)(link)
Thanks for the awesome response time! <3
kuwdora: Pooka - card 60, brian froud (Default)

[personal profile] kuwdora 2013-01-13 10:54 pm (UTC)(link)
♥ !
senmut: modern style black canary on right in front of modern style deathstroke (Default)

[personal profile] senmut 2013-01-13 10:56 pm (UTC)(link)
Dang but you all rock for that response.
ilyena_sylph: picture of Labyrinth!faerie with 'careful, i bite' as text (Default)

[personal profile] ilyena_sylph 2013-01-13 10:57 pm (UTC)(link)
+1.

I was driving so I didn't even see it.

I got home and there were announcements that DW was down, that you were working on it, and then within seconds of my getting home it was all better!

<3333333333333
jumble: (back)

[personal profile] jumble 2013-01-13 10:58 pm (UTC)(link)
Seriously, props on not just the response time but the communication. It is bar none. This is a big part of why I love DW.
dragondancer5150: (Default)

[personal profile] dragondancer5150 2013-01-13 11:06 pm (UTC)(link)
SECONDING

(no subject)

[personal profile] jelazakazone - 2013-01-13 23:21 (UTC) - Expand

(no subject)

[personal profile] alionheartedgirl - 2013-01-14 06:19 (UTC) - Expand

(no subject)

[personal profile] narrativian - 2013-01-16 00:02 (UTC) - Expand
dil: (Default)

[personal profile] dil 2013-01-13 10:59 pm (UTC)(link)
Monitoring free space would be a good idea :)

Anyhow thanks for the perfect response time!
alierak: (Default)

[personal profile] alierak 2013-01-13 11:10 pm (UTC)(link)
Yeah. We also realized we've been graphing but not alerting on disk space :/
zing_och: Grace Choi from the Outsiders comic (Default)

[personal profile] zing_och 2013-01-13 11:00 pm (UTC)(link)
Wow, that was fast! Thank you for letting us know.
lawless523: kanzeon bosatsu (Default)

[personal profile] lawless523 2013-01-13 11:00 pm (UTC)(link)
I just tried posting a comment on someone else's journal and wasn't able to.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2013-01-13 11:09 pm (UTC)(link)
When you say "not able to", what happened?

(no subject)

[personal profile] lawless523 - 2013-01-13 23:14 (UTC) - Expand

(no subject)

[staff profile] denise - 2013-01-13 23:18 (UTC) - Expand

(no subject)

[personal profile] dil - 2013-01-13 23:40 (UTC) - Expand

(no subject)

[staff profile] denise - 2013-01-13 23:42 (UTC) - Expand
(screened comment)

(frozen comment) (no subject)

[staff profile] denise - 2013-01-14 00:37 (UTC) - Expand

(no subject)

[personal profile] lawless523 - 2013-01-14 00:29 (UTC) - Expand

(no subject)

[staff profile] denise - 2013-01-14 00:33 (UTC) - Expand
hellkitty: (kitten flattened)

[personal profile] hellkitty 2013-01-13 11:00 pm (UTC)(link)
You guys are amazing. Things happen but you were on it immediately *and* communicated with us fast and honestly.
strega_lyth: миру - мир (Default)

[personal profile] strega_lyth 2013-01-13 11:01 pm (UTC)(link)
Thanks!
pretty_panther: (av: thor and natasha)

[personal profile] pretty_panther 2013-01-13 11:02 pm (UTC)(link)
Dayum you guys were quick! Really appreciate your work and keeping us posted!
solmedes: (daughter (pm2) √ knight's honor)

[personal profile] solmedes 2013-01-13 11:04 pm (UTC)(link)
Seconding what they all said. That was remarkably quick.

I'm glad it was only disk space, and not something more serious.
darjeeling: (Default)

[personal profile] darjeeling 2013-01-13 11:04 pm (UTC)(link)
Not only was the hiccup barely long enough to even register as downtime, you swarmed all over fixing it like a Zergling rush AND we even got technical follow up on what caused it.

You guys are too awesome for words. ♥
mn: (mn smile)

[personal profile] mn 2013-01-13 11:11 pm (UTC)(link)
Thank you very much indeed for fixing this so quickly!
darth_eldritch: (Black Hole)

[personal profile] darth_eldritch 2013-01-13 11:12 pm (UTC)(link)
Thank you!

I knew everything was going to be fine soon!
justhuman: dreamwidth icons with paul gross arms :-) (dreamwidth-yay)

[personal profile] justhuman 2013-01-13 11:34 pm (UTC)(link)
Thanks for the quick response and even more for the update. We love that you folks talk to us.
darth_eldritch: (Galaxies)

[personal profile] darth_eldritch 2013-01-13 11:47 pm (UTC)(link)
I can't get on my journal page, except through links to specific entries in my inbox or through tags.

I get the maintainence page when I click on my username, or any link that goes directly to my journal, such as from my favorites bar.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2013-01-13 11:53 pm (UTC)(link)
Can you try clearing your browser cache? Most likely, your browser is holding onto a cached copy of the page.

(no subject)

[personal profile] darth_eldritch - 2013-01-14 01:10 (UTC) - Expand

[personal profile] refno619012 2013-01-13 11:59 pm (UTC)(link)
Didn't have any issues whatsoever. :D
ofearthandstars: A painted tree, art by Natasha Westcoat (Default)

[personal profile] ofearthandstars 2013-01-14 12:03 am (UTC)(link)
Thanks so much for the clear and quick response! ♥!
author_by_night: (Default)

[personal profile] author_by_night 2013-01-14 12:15 am (UTC)(link)
Still unable to access...

(no subject)

[personal profile] author_by_night - 2013-01-14 00:21 (UTC) - Expand
the: (g. hedlund ☛ ( what it says on the tin ))

[personal profile] the 2013-01-14 12:20 am (UTC)(link)
thanks dudes! tho when i try to go to [community profile] rpanons (not the reading page or profile, the main page) i still get "We're sorry, Dreamwidth Studios is currently undergoing maintenance or downtime..."
the: (m. fassbender ☛ ( define tender ))

[personal profile] the 2013-01-14 12:21 am (UTC)(link)
aaand now it's gone!

(no subject)

[staff profile] denise - 2013-01-14 00:23 (UTC) - Expand

(no subject)

[personal profile] the - 2013-01-14 02:27 (UTC) - Expand
peoppenheimer: A photo of Paul Oppenheimer at the Australasian Association of Philosophy meeting. (Default)

[personal profile] peoppenheimer 2013-01-14 12:51 am (UTC)(link)
Mad props for communication!! (1) Notice went up on Twitter right away. (2) The problem was diagnosed and fixed very quickly. (3) You gave us a detailed and comprehensible explanation, with a plan for a longer-term fix. You rock!
danno_williams: (Default)

[personal profile] danno_williams 2013-01-14 01:11 am (UTC)(link)
Forgive the RP journal.

I love this place. I'm always willing to give organizations and people more leeway if they just have the guts to be forthcoming about problems and what they are doing to fix it.

So mad props and awesome customer service and all that.
sara: S (Default)

[personal profile] sara 2013-01-14 04:33 am (UTC)(link)
You're so courteous to go down while I was having a nap anyway....;>
cybel: (Default)

[personal profile] cybel 2013-01-14 05:57 am (UTC)(link)
Since this morning every time I load a new page at amplificathon.dreamwidth.org a generic file downloads to my Mac. It's just called player, no file type indicated. In Get Info it's just called a document, with a default to open in Text Edit. Needless to say I keep trashing the file without opening.

Any idea what might be causing this?
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2013-01-14 06:09 am (UTC)(link)
Looks like it's your computer or browser trying to download the player for one of the embedded files, probably because of an extension or something?

(no subject)

[personal profile] cybel - 2013-01-14 06:15 (UTC) - Expand

(no subject)

[staff profile] denise - 2013-01-14 06:22 (UTC) - Expand

(no subject)

[personal profile] cybel - 2013-01-14 06:44 (UTC) - Expand

I didn't even notice.

[personal profile] babysprite 2013-01-14 07:35 am (UTC)(link)
Honestly, I didn't notice.

By the way, Dreamwidth looks and works great with the w3m web browser.

Yours from the ba$h shell,
/\__/\
(='.'=)
(")_(")
babysprite
yvi: Kaylee half-smiling, looking very pretty (Default)

Re: I didn't even notice.

[personal profile] yvi 2013-01-14 05:44 pm (UTC)(link)
Oh, someone using a text-browser, awesome!

(I once tried my luck with Livejournal and Lynx when I broke my XServer - I don't remember how well that worked :))

Page 1 of 2