Network service restored

[denise]

At approximately 1AM EST / 6AM GMT, connectivity to the site was lost. It was entirely connectivity-based: our service provider had a networking issue with their own providers. So, our servers were just fine, they just couldn't talk to anybody!

Connectivity appears to have been restored at this point. We're really sorry for the inconvenience.

Comments

[ozqueen]

Thank you for keeping us updated on twitter ♥ You guys are stars.

[green_knight]

You guys are stars

This.

I was fast asleep during the outage, and saw the twitter notices on my stream this morning, I was forewarned, I felt fully informed, and if DW had still been down, it woulnd't have been a problem at all - technology happens.

[kore]

Exactly -- being informed makes a big difference.

[owlmoose]

Very much agreed. Thank you for being communicative and honest! :)

[dancing_serpent]

Yay, DW is back! *clings*

Thank you for keeping us updated on Twitter.

[calvinahobbes]

I woke up, and DW was gone!!!

Thank you for twitter, indeed!

[claidheamhmor]

Welcome back! Thanks for the Twitter updates.

[senmut]

Y'all rocked, though. We knew, and that helps!

[blundergirl]

Thanks for the updates!

You learn something every day...

[sepdet]

...and I just learned that Dreamwidth and Tvtropes must be sharing the same service provider, because you both went down and came back at the same time!

It's a small (fannish) world after all...

Glad you're back!

I admit it felt very surreal posting to LJ because DW was down, instead of the reverse!

[thisfffire]

Thank you so much for keeping us updated and thanks again for letting us know what was going on. You guys rock.

[cmcmck]

Thanks!

I did wonder where you'd gone first thing this morning (UK time).

[kore]

Thank you for updating us so quickly! The Twitter status posts were great.

[dil]

Thanks for keeping us informed.

[lawless523]

Have you sorted through the issue with your service provider so that you're confident this isn't likely to occur again?

[denise]

If you click through to the link in the post, you can see the underlying cause and the steps they're taking to prevent the issue in the future -- it wasn't just us affected, it was the entire data center.

[lawless523]

I clicked through but wasn't able to make much sense of the information or to even to follow it. It gave me no sense of how likely it was to recur.

[denise]

"At the moment it is unclear how separate providers with previously diverse circuits paths would both be impacted by the same fiber work. We will be following up with our San Antonio transport providers and assessing how this scope of work had the impact that it did and then take appropriate action to ensure circuit redundancy is reinstituted. Additionally, PEER 1 Hosting will be doing an audit of the current fiber paths being used throughout our transport links to ensure full network redundancy is available in all locations."

In other words, the datacenter had previously made provisions for redundant circuits from separate providers so that an outage from one provider would not result in the datacenter being unreachable, but due to a not-yet-known cause, the maintenance that was conducted without notice by one of their circuit providers caused the datacenter to be unreachable. They are following up with their providers to determine what caused the problem and how to avoid it in the future.

No provider is able to guarantee 100% uptime, but we've always been extremely satisfied with Peer1's reliability and their responsiveness when issues do happen; they're definitely among the best options out there. In this case, they acted quickly and decisively, and we are confident they will handle the followup appropriately.

[sepdet]

Yep. It sounds like their heads are nowhere near their posteriors, and they're on it.

Perfection I do not expect. Response to problems, that I expect.

[lawless523]

Thank you for the explanation.

[dil]

You can never tell with these modern electronic devices/ They either work or not, but nobody knows for sure why ;)

"The transport circuits that connect Dallas and San Antonio are supplied by different providers, and are specified as running on diverse fiber paths. Continuing our investigation, the Network Operations Center was able to determine that the connectivity loss was the result of a fiber maintenance being performed by an underlying long-haul carrier, for which PEER 1 Hosting was not notified. At 03:30 CT, the first transport circuit came back online, which fully restored network connectivity to the San Antonio Data Center.

At the moment it is unclear how separate providers with previously diverse circuits paths would both be impacted by the same fiber work. We will be following up with our San Antonio transport providers and assessing how this scope of work had the impact that it did and then take appropriate action to ensure circuit redundancy is reinstituted. Additionally, PEER 1 Hosting will be doing an audit of the current fiber paths being used throughout our transport links to ensure full network redundancy is available in all locations."

[azurelunatic]

At the moment it is unclear how separate providers with previously diverse circuits paths would both be impacted by the same fiber work. We will be following up with our San Antonio transport providers and assessing how this scope of work had the impact that it did and then take appropriate action to ensure circuit redundancy is reinstituted.

This sounds very much like an angry network admin with a hammer who is trying to sound professional before starting to hit things.

[brownbetty]

I was going to link the xkcd comic, but his site's gone all wibbledy. (Probably unconnected? Sunspots?) I embed the link here anyway, since it will probably come on-line eventually:

[weofodthignen]

Thanks for letting us know :-) I ran into the problem as I was headed to bed last night and was relieved to see the site back this morning.

M

[seriouslytwisted]

hahaha it's cool i actually got to bed at a decent hour for once because of it

[subluxate]

Lol, same.

Thanks for keeping us updated, both here and on Twitter!

[shoots_kneecaps]

Thanks for the update - I was worried there for a bit! I'd just finished writing an RP tag and hit 'post comment' and *poof* - no DW! I am eternally thankful to whoever created the add-on Lazarus, because it saved my comment and I was able to e-mail it to myself and then post it this morning.

[subluxate]

Ooh, did not know about that add-on; thanks for mentioning it!

[shoots_kneecaps]

It has seriously saved my sanity on a number of occasions. The website seems to be having some problems right now, so I hope you don't have trouble getting it!

[firebird308]

Just be warned that firefox sometimes likes to stop working right if it decides to fight with whatever.

Although yes, go ahead and try it for in case it actually decides to like you and behave.

[subluxate]

Yeah, if I have issues, I'll delete it. But it's worth a shot.

[nightdog_barks]

I didn't know about this either! I could've saved that paragraph of fic last night, dadgummit. Off to install now; thanks for the mention!

[finch]

My home internet is so flakey at the moment that it took me almost half an hour to notice. XD Thanks for the twitter updates so I knew it wasn't just me.

[darth_eldritch]

Huh? Wat? What happened?

Never noticed a thing. Which shows how awesome you guys are.

[ardath_rekha]

You were such an enormous help to me last night (thank you so much!), and then this hit you soon after. Wow! I'd have been tearing my hair out. You are all so awesome. :)

[moem]

I'm glad y'all are back!

[random_xtras]

And I completely missed it. I'm so glad. I'm busy freaking about other stuff right now and didn't need another thing on the list. LOL!

[sami]

I hope they weren't lonely. :(

[denise]

Well, they could talk to each other! just not the internet. so it was more like everybody standing around going "i'm bored. are you bored? yeah, i'm bored."

[azurelunatic]

I wonder -- in case something like this happens again (or even in cases of Severe Just Nothing Happening) -- if there's a way to have the servers notice "Hey, we've been pretty dang idle for about five minutes. Maybe we could do some expensive internal stuff that doesn't really involve a lot of mail." Something like an expunge-very-old-deleted-users run? (Though I think you tend to give a courtesy heads-up in a news update before those get run, often...)

I'm thinking an internal activity monitor that notices when load dips below a certain threshold for a certain length of time, and then starts doing some queued crunching that might have otherwise been saved for a maintenance window?

[shadowchaser]

I seriously love you guys for the way you keep us informed. Thank you.

[dickgrayson]

Thank you <3