denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
Denise ([staff profile] denise) wrote in [site community profile] dw_maintenance2011-08-19 07:04 am
  • Add Memory
  • Share This Entry

This morning's downtime

The downtime we had earlier this morning was due to a server upgrade at our hosting company -- we asked them to upgrade one of our machines from 4GB of RAM to 8GB of RAM, and to do it this time between 5AM and 6AM CDT (their timezone) with 12 hours' notice so we could let y'all know about the planned downtime. Well, they caught the requested window of upgrade time this time, but they missed the "with 12 hours' notice" part!

The machine that was taken out of the pool in order to upgrade happened to be a critical part of our infrastructure (the load balancer -- the machine that "listens" for traffic and directs it to the appropriate places on the network). We have our servers set up so that an outage of one machine, even if it's the load balancer, shouldn't cause the site to be completely down, but this morning exposed a glitch in our configuration that caused the "failover" to the backup load balancer to not kick in the way it was supposed to.

So, once the upgrade was complete, [personal profile] alierak (who is rocking the sysadmin stuff this month) took some time out of his hectic morning to make sure everything went back to running along nicely so I didn't have to wake up [staff profile] mark in the middle of his night, and we'll do what we can to make sure that in the future, losing the main load balancer will cause traffic to fail over to our backup load balancer more seamlessly.

Thanks, as always, to y'all for your patience with us, and for your continued support!
Flat | Top-Level Comments Only
cesy: "Cesy" - An old-fashioned quill and ink (Default)

[personal profile] cesy 2011-08-19 12:02 pm (UTC)(link)
Thank you for posting to Twitter so promptly and explaining everything afterwards.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2011-08-19 12:09 pm (UTC)(link)
Fortunately it happened when I was awake and in front of my computer. *g*
dreamatdrew: An orange leopard gecko half hiding behind the leaf of a 'lucky bamboo' plant, looking directly at you. (Default)

[personal profile] dreamatdrew 2011-08-19 01:01 pm (UTC)(link)
Dude, they are just FAILING with the notifications bit aren't they

[personal profile] rowrbazzle 2011-08-19 03:41 pm (UTC)(link)
Seriously. Hosting company needs a good smack with the clue-by-four.

[personal profile] ficwriter1966 2011-08-19 01:05 pm (UTC)(link)
Thank you so much for keeping us informed! I was probably asleep when it happened, but I imagine some folks were a little taken aback. Hope things go more smoothly from now on - and have a great weekend, you admin folk!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2011-08-19 01:15 pm (UTC)(link)
Heh. I haven't taken a 'weekend' in months! But I'm on vacation for five whole glorious days starting next week, and right now I'm thinking that I'll leave the laptop at home :)
urbanfuturistic: (It the one without a face.)

[personal profile] urbanfuturistic 2011-08-19 09:10 pm (UTC)(link)
That would be sensible... which is why I never do it.
janewilliams20: (Default)

[personal profile] janewilliams20 2011-08-19 01:18 pm (UTC)(link)
Who is this hosting company? Not that I'd use one by choice in either "CDT" (having looked it up, while wondering what's wrong with giving an actual timezone, it seems to be either UTC-05 or UTC+10:30), but it would be nice to know who to avoid.

pauamma: Cartooney crab wearing hot pink and acid green facemask holding drink with straw (Default)

[personal profile] pauamma 2011-08-19 03:14 pm (UTC)(link)
It's most likely the US CDT, not the Australian one. :-) So UTC-5 hours.
bubbleblower: cropped head shot of me with nebula background (Default)

[personal profile] bubbleblower 2011-08-19 04:51 pm (UTC)(link)
It's probably UTC-5, but there's still a valid point.

People in other countries may not know the US time zones, and the names may be ambiguous if two different countries use the same names for different zones.

At the same time, many Americans don't know what UTC is or what the numbers mean.

The best policy may be to give it in both formats, as in "CDT (UTC-5)". There will still be some who won't be able to figure it out, but hopefully most will.
pauamma: Cartooney crab wearing hot pink and acid green facemask holding drink with straw (Default)

[personal profile] pauamma 2011-08-19 05:32 pm (UTC)(link)
True.
alierak: (Default)

[personal profile] alierak 2011-08-19 07:37 pm (UTC)(link)
Maybe I'll try that next time. I'm in Alabama and the datacenter is in Texas, so I would have assumed we were in the same timezone and speaking the same language. But maybe there's an office somewhere else where they receive all the requests first and make an effort to misunderstand them...
rpdom: Me wearing my first pair of reading glasses (Default)

[personal profile] rpdom 2011-08-20 04:05 am (UTC)(link)
I find the "Event Time Announcer" at http://www.timeanddate.com/worldclock/fixedform.html very useful for this sort of thing.

Put in details (time, date, timezone) of your event and it returns a page with worldwide local times. Paste the resulting url into your post.

Eg: Unexpected memory upgrade

No confusion there :-)

(Still doesn't help when the hosting company decides to do it without notice though)
intermezzo: (Default)

[personal profile] intermezzo 2011-08-19 02:20 pm (UTC)(link)
thank you for the explanation via Twitter. :)
blessed_oak: (Default)

[personal profile] blessed_oak 2011-08-19 02:54 pm (UTC)(link)
I was asleep too.

But really I just wanted to say that I'm new here and really like this place! :)
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2011-08-19 02:57 pm (UTC)(link)
Welcome! It sounds cheesy, I know, but I'm really glad to have you here (even though I don't know you -- I always, always love to see new people!) and I hope you find DW to be an awesome home for you :)
gwendraith: (mouse bow)

[personal profile] gwendraith 2011-08-19 04:05 pm (UTC)(link)
Thanks for the explanation. I almost had a panic attack when DW disappeared but I knew there'd be a sensible explanation and it'd be back soon :) After 8 years on LJ I'm really liking it here and am making friends easily :)
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2011-08-19 04:07 pm (UTC)(link)
Welcome to Dreamwidth! I'm so glad to have you here :)
(deleted comment)
gwendraith: (twilight)

[personal profile] gwendraith 2011-08-19 08:15 pm (UTC)(link)
Thank you! Mouse is my online name in most places. I see you are a twilight fan. Me too, anything vampirey or gothy, although I'm old and should know better ;)
calvinahobbes: Calvin holding a cardboard tv-shape up in front of himself (dreamwidthsheep)

[personal profile] calvinahobbes 2011-08-19 04:06 pm (UTC)(link)
Well, it was my middle-of-the-afternoon and I was really sad to find DW suddenly unresponsive D:

But, thanks to a prompt tweet I knew what was going on right away, and thanks to DW's awesomeness I could just back-button, and the comment I was trying to post was still there \o/

I'm glad things should be fixed for now.
rpdom: Me wearing my first pair of reading glasses (Default)

[personal profile] rpdom 2011-08-19 04:47 pm (UTC)(link)
It was mid-morning here when [profile] gipsy_dreamer turned to me and said "Is the internet down?". I checked and replied "No, only Dreamwidth. They've probably had another memory upgrade with no notice".

Seems I was right :-)

Good to have you back!

(Hosting company need serious prodding)
alierak: (Default)

[personal profile] alierak 2011-08-20 02:34 pm (UTC)(link)
Sounds like they're going to give us one of the upgrades for free.
goodbyebird: Batman returns: Catwoman seen through a glass window. (Default)

[personal profile] goodbyebird 2011-08-20 12:41 am (UTC)(link)
Thanks for keeping us in the loop, and for your hard work <3
soundingsea: (Default)

[personal profile] soundingsea 2011-08-20 02:06 am (UTC)(link)
One of the things I like most about Dreamwidth is your reasonable explanations. That's why I bought a seed account and keep inviting people here; I trust that you know what you're doing.
geminianeyes: Cute sisters from PW as kids (Default)

[personal profile] geminianeyes 2011-08-20 04:36 am (UTC)(link)
It was nearly night for me. Was trying to update a blog and realised oh, DW's down. Then I went out for dinner, cause I knew that when it came back I'd see the DW message telling us what went wrong.

You guys are awesome. <3
azurelunatic: A red shield with a golden-orange phoenix. Caption: "House of hEll" (hEll)

[personal profile] azurelunatic 2011-08-20 08:17 am (UTC)(link)
I am so happy with [personal profile] sophie's little bots. Especially the "HEY MARK HEY MARK HEY MARK" bit.
frith: Violet unicorn cartoon pony with a blue mane (FIM Twilight friendly)

[personal profile] frith 2011-08-20 12:28 pm (UTC)(link)
Neat! While I don't have the background to comprehend the nittygritty details, I get the overall picture. 8^) Unending thrills on the Rube Goldberg servers! These things seem so complicated... there must be a module on it somewhere that prods the octopus that squeezes the orange that fills the glass that slides into [staff profile] mark's breakfast tray in the morning. 8^D

Wow! Only 8 gig of RAM. Cool. 8^)
milkymoon: (HELLO THAR.)

[personal profile] milkymoon 2011-08-22 04:55 pm (UTC)(link)
You guys are awesome for keeping your customers in the loop. :)
Flat | Top-Level Comments Only