Network service restored

[denise]

At approximately 1AM EST / 6AM GMT, connectivity to the site was lost. It was entirely connectivity-based: our service provider had a networking issue with their own providers. So, our servers were just fine, they just couldn't talk to anybody!

Connectivity appears to have been restored at this point. We're really sorry for the inconvenience.

Comments

[denise]

If you click through to the link in the post, you can see the underlying cause and the steps they're taking to prevent the issue in the future -- it wasn't just us affected, it was the entire data center.

[lawless523]

I clicked through but wasn't able to make much sense of the information or to even to follow it. It gave me no sense of how likely it was to recur.

[denise]

"At the moment it is unclear how separate providers with previously diverse circuits paths would both be impacted by the same fiber work. We will be following up with our San Antonio transport providers and assessing how this scope of work had the impact that it did and then take appropriate action to ensure circuit redundancy is reinstituted. Additionally, PEER 1 Hosting will be doing an audit of the current fiber paths being used throughout our transport links to ensure full network redundancy is available in all locations."

In other words, the datacenter had previously made provisions for redundant circuits from separate providers so that an outage from one provider would not result in the datacenter being unreachable, but due to a not-yet-known cause, the maintenance that was conducted without notice by one of their circuit providers caused the datacenter to be unreachable. They are following up with their providers to determine what caused the problem and how to avoid it in the future.

No provider is able to guarantee 100% uptime, but we've always been extremely satisfied with Peer1's reliability and their responsiveness when issues do happen; they're definitely among the best options out there. In this case, they acted quickly and decisively, and we are confident they will handle the followup appropriately.

[sepdet]

Yep. It sounds like their heads are nowhere near their posteriors, and they're on it.

Perfection I do not expect. Response to problems, that I expect.

[lawless523]

Thank you for the explanation.

[dil]

You can never tell with these modern electronic devices/ They either work or not, but nobody knows for sure why ;)

"The transport circuits that connect Dallas and San Antonio are supplied by different providers, and are specified as running on diverse fiber paths. Continuing our investigation, the Network Operations Center was able to determine that the connectivity loss was the result of a fiber maintenance being performed by an underlying long-haul carrier, for which PEER 1 Hosting was not notified. At 03:30 CT, the first transport circuit came back online, which fully restored network connectivity to the San Antonio Data Center.

At the moment it is unclear how separate providers with previously diverse circuits paths would both be impacted by the same fiber work. We will be following up with our San Antonio transport providers and assessing how this scope of work had the impact that it did and then take appropriate action to ensure circuit redundancy is reinstituted. Additionally, PEER 1 Hosting will be doing an audit of the current fiber paths being used throughout our transport links to ensure full network redundancy is available in all locations."

[azurelunatic]

At the moment it is unclear how separate providers with previously diverse circuits paths would both be impacted by the same fiber work. We will be following up with our San Antonio transport providers and assessing how this scope of work had the impact that it did and then take appropriate action to ensure circuit redundancy is reinstituted.

This sounds very much like an angry network admin with a hammer who is trying to sound professional before starting to hit things.

[brownbetty]

I was going to link the xkcd comic, but his site's gone all wibbledy. (Probably unconnected? Sunspots?) I embed the link here anyway, since it will probably come on-line eventually: