denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
Denise ([staff profile] denise) wrote in [site community profile] dw_maintenance2013-02-12 04:39 am

Network service restored

At approximately 1AM EST / 6AM GMT, connectivity to the site was lost. It was entirely connectivity-based: our service provider had a networking issue with their own providers. So, our servers were just fine, they just couldn't talk to anybody!

Connectivity appears to have been restored at this point. We're really sorry for the inconvenience.
ozqueen: (Default)

[personal profile] ozqueen 2013-02-12 09:50 am (UTC)(link)
Thank you for keeping us updated on twitter ♥ You guys are stars.
green_knight: (Bravo)

[personal profile] green_knight 2013-02-12 10:35 am (UTC)(link)
You guys are stars

This.

I was fast asleep during the outage, and saw the twitter notices on my stream this morning, I was forewarned, I felt fully informed, and if DW had still been down, it woulnd't have been a problem at all - technology happens.
kore: (Default)

[personal profile] kore 2013-02-12 01:51 pm (UTC)(link)
Exactly -- being informed makes a big difference.
owlmoose: (cats - silver kitty)

[personal profile] owlmoose 2013-02-12 07:33 pm (UTC)(link)
Very much agreed. Thank you for being communicative and honest! :)
dancing_serpent: (Default)

[personal profile] dancing_serpent 2013-02-12 09:51 am (UTC)(link)
Yay, DW is back! *clings*

Thank you for keeping us updated on Twitter.
calvinahobbes: happy seal (seal)

[personal profile] calvinahobbes 2013-02-12 12:05 pm (UTC)(link)
I woke up, and DW was gone!!!

Thank you for twitter, indeed!
claidheamhmor: (Default)

[personal profile] claidheamhmor 2013-02-12 09:57 am (UTC)(link)
Welcome back! Thanks for the Twitter updates.
sharpest_asp: Close up of a lavender eye in a dark face (Default)

[personal profile] sharpest_asp 2013-02-12 10:00 am (UTC)(link)
Y'all rocked, though. We knew, and that helps!
blundergirl: SUCK IT LAGOON BOY. (♥ » OH yeah! ALPHA squad!)

[personal profile] blundergirl 2013-02-12 10:22 am (UTC)(link)
Thanks for the updates!
sepdet: (hurtthem)

You learn something every day...

[personal profile] sepdet 2013-02-12 10:22 am (UTC)(link)
...and I just learned that Dreamwidth and Tvtropes must be sharing the same service provider, because you both went down and came back at the same time!

It's a small (fannish) world after all...

Glad you're back!

I admit it felt very surreal posting to LJ because DW was down, instead of the reverse!
thisfffire: I WONDER HOW I CAN TALK YOU OUT OF EVER MAKING THAT FACE AGAIN... (boy you got a pretty mouth)

[personal profile] thisfffire 2013-02-12 11:58 am (UTC)(link)
Thank you so much for keeping us updated and thanks again for letting us know what was going on. You guys rock.
cmcmck: (Default)

[personal profile] cmcmck 2013-02-12 01:20 pm (UTC)(link)
Thanks!

I did wonder where you'd gone first thing this morning (UK time).
kore: (Default)

[personal profile] kore 2013-02-12 01:50 pm (UTC)(link)
Thank you for updating us so quickly! The Twitter status posts were great.
dil: (Default)

[personal profile] dil 2013-02-12 02:31 pm (UTC)(link)
Thanks for keeping us informed.
lawless523: (gravitation: eiri on the internets)

[personal profile] lawless523 2013-02-12 04:16 pm (UTC)(link)
Have you sorted through the issue with your service provider so that you're confident this isn't likely to occur again?
lawless523: (gravitation: eiri on the internets)

[personal profile] lawless523 2013-02-12 04:33 pm (UTC)(link)
I clicked through but wasn't able to make much sense of the information or to even to follow it. It gave me no sense of how likely it was to recur.
sepdet: Samhain worshipping the veggies. Oooommm. (Okay, yes, catnip was involved.) (Default)

[personal profile] sepdet 2013-02-12 09:37 pm (UTC)(link)
Yep. It sounds like their heads are nowhere near their posteriors, and they're on it.

Perfection I do not expect. Response to problems, that I expect.
lawless523: (Default)

[personal profile] lawless523 2013-02-13 02:42 pm (UTC)(link)
Thank you for the explanation.
dil: (Default)

[personal profile] dil 2013-02-12 04:54 pm (UTC)(link)
You can never tell with these modern electronic devices/ They either work or not, but nobody knows for sure why ;)

"The transport circuits that connect Dallas and San Antonio are supplied by different providers, and are specified as running on diverse fiber paths. Continuing our investigation, the Network Operations Center was able to determine that the connectivity loss was the result of a fiber maintenance being performed by an underlying long-haul carrier, for which PEER 1 Hosting was not notified. At 03:30 CT, the first transport circuit came back online, which fully restored network connectivity to the San Antonio Data Center.

At the moment it is unclear how separate providers with previously diverse circuits paths would both be impacted by the same fiber work. We will be following up with our San Antonio transport providers and assessing how this scope of work had the impact that it did and then take appropriate action to ensure circuit redundancy is reinstituted. Additionally, PEER 1 Hosting will be doing an audit of the current fiber paths being used throughout our transport links to ensure full network redundancy is available in all locations."
azurelunatic: Azz and best friend grabbing each other's noses.  (Default)

[personal profile] azurelunatic 2013-02-12 07:47 pm (UTC)(link)
At the moment it is unclear how separate providers with previously diverse circuits paths would both be impacted by the same fiber work. We will be following up with our San Antonio transport providers and assessing how this scope of work had the impact that it did and then take appropriate action to ensure circuit redundancy is reinstituted.

This sounds very much like an angry network admin with a hammer who is trying to sound professional before starting to hit things.
brownbetty: (Default)

[personal profile] brownbetty 2013-02-12 09:36 pm (UTC)(link)
I was going to link the xkcd comic, but his site's gone all wibbledy. (Probably unconnected? Sunspots?) I embed the link here anyway, since it will probably come on-line eventually:

weofodthignen: selfportrait with Rune the cat (Default)

[personal profile] weofodthignen 2013-02-12 05:00 pm (UTC)(link)
Thanks for letting us know :-) I ran into the problem as I was headed to bed last night and was relieved to see the site back this morning.

M
seriouslytwisted: (watching leo screw up is AWESOME.)

[personal profile] seriouslytwisted 2013-02-12 05:18 pm (UTC)(link)
hahaha it's cool i actually got to bed at a decent hour for once because of it
subluxate: Sophia Bush leaning against a piano (Default)

[personal profile] subluxate 2013-02-12 06:17 pm (UTC)(link)
Lol, same.

Thanks for keeping us updated, both here and on Twitter!
shoots_kneecaps: (OOC)

[personal profile] shoots_kneecaps 2013-02-12 06:17 pm (UTC)(link)
Thanks for the update - I was worried there for a bit! I'd just finished writing an RP tag and hit 'post comment' and *poof* - no DW! I am eternally thankful to whoever created the add-on Lazarus, because it saved my comment and I was able to e-mail it to myself and then post it this morning.
subluxate: Sophia Bush leaning against a piano (Default)

[personal profile] subluxate 2013-02-12 06:20 pm (UTC)(link)
Ooh, did not know about that add-on; thanks for mentioning it!
shoots_kneecaps: (OOC)

[personal profile] shoots_kneecaps 2013-02-12 06:25 pm (UTC)(link)
It has seriously saved my sanity on a number of occasions. The website seems to be having some problems right now, so I hope you don't have trouble getting it!
firebird308: (Default)

[personal profile] firebird308 2013-02-12 07:44 pm (UTC)(link)
Just be warned that firefox sometimes likes to stop working right if it decides to fight with whatever.

Although yes, go ahead and try it for in case it actually decides to like you and behave.
subluxate: Sophia Bush leaning against a piano (Default)

[personal profile] subluxate 2013-02-12 07:47 pm (UTC)(link)
Yeah, if I have issues, I'll delete it. But it's worth a shot.
nightdog_barks: (Bucked Off (by Alex Nabaum))

[personal profile] nightdog_barks 2013-02-12 09:01 pm (UTC)(link)
I didn't know about this either! I could've saved that paragraph of fic last night, dadgummit. Off to install now; thanks for the mention!
finch: "And everything changes and nothing is truly lost" (Default)

[personal profile] finch 2013-02-12 08:14 pm (UTC)(link)
My home internet is so flakey at the moment that it took me almost half an hour to notice. XD Thanks for the twitter updates so I knew it wasn't just me.
darth_eldritch: (Magic Circles)

[personal profile] darth_eldritch 2013-02-12 11:13 pm (UTC)(link)
Huh? Wat? What happened?

Never noticed a thing. Which shows how awesome you guys are.
ardath_rekha: (Default)

[personal profile] ardath_rekha 2013-02-13 03:07 am (UTC)(link)
You were such an enormous help to me last night (thank you so much!), and then this hit you soon after. Wow! I'd have been tearing my hair out. You are all so awesome. :)
moem: A computer drawing that looks like me. (Default)

[personal profile] moem 2013-02-13 09:05 am (UTC)(link)
I'm glad y'all are back!
random_xtras: The words Tam I am on a purple background (OOC)

[personal profile] random_xtras 2013-02-13 05:21 pm (UTC)(link)
And I completely missed it. I'm so glad. I'm busy freaking about other stuff right now and didn't need another thing on the list. LOL!
sami: (Default)

[personal profile] sami 2013-02-14 12:07 am (UTC)(link)
I hope they weren't lonely. :(
azurelunatic: Azz and best friend grabbing each other's noses.  (Default)

[personal profile] azurelunatic 2013-02-16 10:34 pm (UTC)(link)
I wonder -- in case something like this happens again (or even in cases of Severe Just Nothing Happening) -- if there's a way to have the servers notice "Hey, we've been pretty dang idle for about five minutes. Maybe we could do some expensive internal stuff that doesn't really involve a lot of mail." Something like an expunge-very-old-deleted-users run? (Though I think you tend to give a courtesy heads-up in a news update before those get run, often...)

I'm thinking an internal activity monitor that notices when load dips below a certain threshold for a certain length of time, and then starts doing some queued crunching that might have otherwise been saved for a maintenance window?
shadowchaser: (Soundwave sewing)

[personal profile] shadowchaser 2013-02-14 04:22 am (UTC)(link)
I seriously love you guys for the way you keep us informed. Thank you.
starrscream: IDW Spotlight cover April 2010 (Default)

[personal profile] starrscream 2013-02-17 06:59 am (UTC)(link)
Thank you <3