denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (me, standing outside a broken phone booth)Denise ([staff profile] denise) wrote in [site community profile] dw_maintenance,
@ 2013-02-12 04:39 am UTC
  • Previous Entry
  • Add to Memories
  • Tell someone about this!
  • Next Entry
At approximately 1AM EST / 6AM GMT, connectivity to the site was lost. It was entirely connectivity-based: our service provider had a networking issue with their own providers. So, our servers were just fine, they just couldn't talk to anybody!

Connectivity appears to have been restored at this point. We're really sorry for the inconvenience.

(42 comments) - (Post a new comment)
(Flat) (Top-level comments only)

ozqueen: (Pin-up: Travel)


[personal profile] ozqueen
2013-02-12 09:50 am UTC (link)
Thank you for keeping us updated on twitter ♥ You guys are stars.

(Reply to this)  (Thread) 


green_knight: (Bravo)


[personal profile] green_knight
2013-02-12 10:35 am UTC (link)
You guys are stars

This.

I was fast asleep during the outage, and saw the twitter notices on my stream this morning, I was forewarned, I felt fully informed, and if DW had still been down, it woulnd't have been a problem at all - technology happens.

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


kore: (Barbara Cooney - Persephone)


[personal profile] kore
2013-02-12 01:51 pm UTC (link)
Exactly -- being informed makes a big difference.

(Reply to this)  (Thread from start)  (Parent) 


owlmoose: (cats - silver kitty)


[personal profile] owlmoose
2013-02-12 07:33 pm UTC (link)
Very much agreed. Thank you for being communicative and honest! :)

(Reply to this)  (Thread from start)  (Parent) 


dancing_serpent: (YnM - Touda - Sky Serpent)


[personal profile] dancing_serpent
2013-02-12 09:51 am UTC (link)
Yay, DW is back! *clings*

Thank you for keeping us updated on Twitter.

(Reply to this)  (Thread) 


calvinahobbes: happy seal (seal)


[personal profile] calvinahobbes
2013-02-12 12:05 pm UTC (link)
I woke up, and DW was gone!!!

Thank you for twitter, indeed!

(Reply to this)  (Thread from start)  (Parent) 


claidheamhmor: (Claymore-Red)


[personal profile] claidheamhmor
2013-02-12 09:57 am UTC (link)
Welcome back! Thanks for the Twitter updates.

(Reply to this) 


sharpest_asp: Black and white art of a hissing snake arounda dagger (General: Snake Knife)


[personal profile] sharpest_asp
2013-02-12 10:00 am UTC (link)
Y'all rocked, though. We knew, and that helps!

(Reply to this) 


blundergirl: SUCK IT LAGOON BOY. (♥ » OH yeah! ALPHA squad!)


[personal profile] blundergirl
2013-02-12 10:22 am UTC (link)
Thanks for the updates!

(Reply to this) 


sepdet: (hurtthem)

You learn something every day...


[personal profile] sepdet
2013-02-12 10:22 am UTC (link)
...and I just learned that Dreamwidth and Tvtropes must be sharing the same service provider, because you both went down and came back at the same time!

It's a small (fannish) world after all...

Glad you're back!

I admit it felt very surreal posting to LJ because DW was down, instead of the reverse!

(Reply to this) 


thisfffire: I WONDER HOW I CAN TALK YOU OUT OF EVER MAKING THAT FACE AGAIN... (boy you got a pretty mouth)


[personal profile] thisfffire
2013-02-12 11:58 am UTC (link)
Thank you so much for keeping us updated and thanks again for letting us know what was going on. You guys rock.

(Reply to this) 


cmcmck: (lizzie)


[personal profile] cmcmck
2013-02-12 01:20 pm UTC (link)
Thanks!

I did wonder where you'd gone first thing this morning (UK time).

(Reply to this) 


kore: (Barbara Cooney - Persephone)


[personal profile] kore
2013-02-12 01:50 pm UTC (link)
Thank you for updating us so quickly! The Twitter status posts were great.

(Reply to this) 


dil: (pic#883897)


[personal profile] dil
2013-02-12 02:31 pm UTC (link)
Thanks for keeping us informed.

(Reply to this) 


lawless523: kanzeon bosatsu (gravitation: eiri on the internets)


[personal profile] lawless523
2013-02-12 04:16 pm UTC (link)
Have you sorted through the issue with your service provider so that you're confident this isn't likely to occur again?

(Reply to this)  (Thread) 


denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (me, standing outside a broken phone booth)


[staff profile] denise
2013-02-12 04:22 pm UTC (link)
If you click through to the link in the post, you can see the underlying cause and the steps they're taking to prevent the issue in the future -- it wasn't just us affected, it was the entire data center.

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


lawless523: kanzeon bosatsu (gravitation: eiri on the internets)


[personal profile] lawless523
2013-02-12 04:33 pm UTC (link)
I clicked through but wasn't able to make much sense of the information or to even to follow it. It gave me no sense of how likely it was to recur.

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (me, standing outside a broken phone booth)


[staff profile] denise
2013-02-12 04:45 pm UTC (link)
"At the moment it is unclear how separate providers with previously diverse circuits paths would both be impacted by the same fiber work. We will be following up with our San Antonio transport providers and assessing how this scope of work had the impact that it did and then take appropriate action to ensure circuit redundancy is reinstituted. Additionally, PEER 1 Hosting will be doing an audit of the current fiber paths being used throughout our transport links to ensure full network redundancy is available in all locations."


In other words, the datacenter had previously made provisions for redundant circuits from separate providers so that an outage from one provider would not result in the datacenter being unreachable, but due to a not-yet-known cause, the maintenance that was conducted without notice by one of their circuit providers caused the datacenter to be unreachable. They are following up with their providers to determine what caused the problem and how to avoid it in the future.

No provider is able to guarantee 100% uptime, but we've always been extremely satisfied with Peer1's reliability and their responsiveness when issues do happen; they're definitely among the best options out there. In this case, they acted quickly and decisively, and we are confident they will handle the followup appropriately.

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


sepdet: Samhain worshipping the veggies. Oooommm. (Okay, yes, catnip was involved.) (garden)


[personal profile] sepdet
2013-02-12 09:37 pm UTC (link)
Yep. It sounds like their heads are nowhere near their posteriors, and they're on it.

Perfection I do not expect. Response to problems, that I expect.

(Reply to this)  (Thread from start)  (Parent) 


lawless523: kanzeon bosatsu (saiyuki: kanzeon peaches)


[personal profile] lawless523
2013-02-13 02:42 pm UTC (link)
Thank you for the explanation.

(Reply to this)  (Thread from start)  (Parent) 


dil: (pic#883897)


[personal profile] dil
2013-02-12 04:54 pm UTC (link)
You can never tell with these modern electronic devices/ They either work or not, but nobody knows for sure why ;)

"The transport circuits that connect Dallas and San Antonio are supplied by different providers, and are specified as running on diverse fiber paths. Continuing our investigation, the Network Operations Center was able to determine that the connectivity loss was the result of a fiber maintenance being performed by an underlying long-haul carrier, for which PEER 1 Hosting was not notified. At 03:30 CT, the first transport circuit came back online, which fully restored network connectivity to the San Antonio Data Center.

At the moment it is unclear how separate providers with previously diverse circuits paths would both be impacted by the same fiber work. We will be following up with our San Antonio transport providers and assessing how this scope of work had the impact that it did and then take appropriate action to ensure circuit redundancy is reinstituted. Additionally, PEER 1 Hosting will be doing an audit of the current fiber paths being used throughout our transport links to ensure full network redundancy is available in all locations."

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


azurelunatic: cameo-like portrait of <user name="azurelunatic"> in short blue hair. (_support, cameo)


[personal profile] azurelunatic
2013-02-12 07:47 pm UTC (link)
At the moment it is unclear how separate providers with previously diverse circuits paths would both be impacted by the same fiber work. We will be following up with our San Antonio transport providers and assessing how this scope of work had the impact that it did and then take appropriate action to ensure circuit redundancy is reinstituted.

This sounds very much like an angry network admin with a hammer who is trying to sound professional before starting to hit things.

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


brownbetty: (teapot)


[personal profile] brownbetty
2013-02-12 09:36 pm UTC (link)
I was going to link the xkcd comic, but his site's gone all wibbledy. (Probably unconnected? Sunspots?) I embed the link here anyway, since it will probably come on-line eventually:

(Reply to this)  (Thread from start)  (Parent) 


weofodthignen: selfportrait with Rune the cat (default)


[personal profile] weofodthignen
2013-02-12 05:00 pm UTC (link)
Thanks for letting us know :-) I ran into the problem as I was headed to bed last night and was relieved to see the site back this morning.

M

(Reply to this) 


seriouslytwisted: (watching leo screw up is AWESOME.)


[personal profile] seriouslytwisted
2013-02-12 05:18 pm UTC (link)
hahaha it's cool i actually got to bed at a decent hour for once because of it

(Reply to this)  (Thread) 


subluxate: Sophia Bush leaning against a piano (Corlionis: Teresa)


[personal profile] subluxate
2013-02-12 06:17 pm UTC (link)
Lol, same.

Thanks for keeping us updated, both here and on Twitter!

(Reply to this)  (Thread from start)  (Parent) 


shoots_kneecaps: (OOC)


[personal profile] shoots_kneecaps
2013-02-12 06:17 pm UTC (link)
Thanks for the update - I was worried there for a bit! I'd just finished writing an RP tag and hit 'post comment' and *poof* - no DW! I am eternally thankful to whoever created the add-on Lazarus, because it saved my comment and I was able to e-mail it to myself and then post it this morning.

(Reply to this)  (Thread) 


subluxate: Sophia Bush leaning against a piano (Corlionis: Teresa)


[personal profile] subluxate
2013-02-12 06:20 pm UTC (link)
Ooh, did not know about that add-on; thanks for mentioning it!

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


shoots_kneecaps: (OOC)


[personal profile] shoots_kneecaps
2013-02-12 06:25 pm UTC (link)
It has seriously saved my sanity on a number of occasions. The website seems to be having some problems right now, so I hope you don't have trouble getting it!

(Reply to this)  (Thread from start)  (Parent) 


firebird308: (Default)


[personal profile] firebird308
2013-02-12 07:44 pm UTC (link)
Just be warned that firefox sometimes likes to stop working right if it decides to fight with whatever.

Although yes, go ahead and try it for in case it actually decides to like you and behave.

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


subluxate: Sophia Bush leaning against a piano (Corlionis: Teresa)


[personal profile] subluxate
2013-02-12 07:47 pm UTC (link)
Yeah, if I have issues, I'll delete it. But it's worth a shot.

(Reply to this)  (Thread from start)  (Parent) 


nightdog_barks: (Bucked Off (by Alex Nabaum))


[personal profile] nightdog_barks
2013-02-12 09:01 pm UTC (link)
I didn't know about this either! I could've saved that paragraph of fic last night, dadgummit. Off to install now; thanks for the mention!

(Reply to this)  (Thread from start)  (Parent) 


finch: drawing of a redheaded white guy with glasses (Lost: Robin in glasses)


[personal profile] finch
2013-02-12 08:14 pm UTC (link)
My home internet is so flakey at the moment that it took me almost half an hour to notice. XD Thanks for the twitter updates so I knew it wasn't just me.

(Reply to this) 


darth_eldritch: (Magic Circles)


[personal profile] darth_eldritch
2013-02-12 11:13 pm UTC (link)
Huh? Wat? What happened?

Never noticed a thing. Which shows how awesome you guys are.

(Reply to this) 


ardath_rekha: (Fairy Wings, Fairy Wings (blue), Fairy Wings (Blue))


[personal profile] ardath_rekha
2013-02-13 03:07 am UTC (link)
You were such an enormous help to me last night (thank you so much!), and then this hit you soon after. Wow! I'd have been tearing my hair out. You are all so awesome. :)

(Reply to this) 


moem: A computer drawing that looks like me. (default)


[personal profile] moem
2013-02-13 09:05 am UTC (link)
I'm glad y'all are back!

(Reply to this) 


random_xtras: The words Tam I am on a purple background (OOC)


[personal profile] random_xtras
2013-02-13 05:21 pm UTC (link)
And I completely missed it. I'm so glad. I'm busy freaking about other stuff right now and didn't need another thing on the list. LOL!

(Reply to this) 


sami: (arise, gaming gamer who games)


[personal profile] sami
2013-02-14 12:07 am UTC (link)
I hope they weren't lonely. :(

(Reply to this)  (Thread) 


denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (me, standing outside a broken phone booth)


[staff profile] denise
2013-02-14 12:13 am UTC (link)
Well, they could talk to each other! just not the internet. so it was more like everybody standing around going "i'm bored. are you bored? yeah, i'm bored."

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


azurelunatic: cameo-like portrait of <user name="azurelunatic"> in short blue hair. (_support, cameo)


[personal profile] azurelunatic
2013-02-16 10:34 pm UTC (link)
I wonder -- in case something like this happens again (or even in cases of Severe Just Nothing Happening) -- if there's a way to have the servers notice "Hey, we've been pretty dang idle for about five minutes. Maybe we could do some expensive internal stuff that doesn't really involve a lot of mail." Something like an expunge-very-old-deleted-users run? (Though I think you tend to give a courtesy heads-up in a news update before those get run, often...)

I'm thinking an internal activity monitor that notices when load dips below a certain threshold for a certain length of time, and then starts doing some queued crunching that might have otherwise been saved for a maintenance window?

(Reply to this)  (Thread from start)  (Parent) 


shadowchaser: (Soundwave sewing)


[personal profile] shadowchaser
2013-02-14 04:22 am UTC (link)
I seriously love you guys for the way you keep us informed. Thank you.

(Reply to this) 


starrscream: (Transformers g1 fan art) (Starscream g1 → chibi just evil)


[personal profile] starrscream
2013-02-17 06:59 am UTC (link)
Thank you <3

(Reply to this) 


(42 comments) - (Post a new comment)
(Flat) (Top-level comments only)