denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
Denise ([staff profile] denise) wrote in [site community profile] dw_maintenance 2017-02-28 06:00 am (UTC)

Disclaimer: I am not the person who handles this stuff, so this is "to the best of my knowledge" and may be slightly out of date or inaccurate!

Pretty much everything is redundant! The webservers are multiply redundant (they go in and out all the time because of load, runaway processes, etc, and y'all never notice), and we have a failover database server that gets all site activity replicated to it automatically so it's ready and waiting to take over. (I can't remember if it's an automatic failover or if somebody would have to flip the switch to activate it. I think it's flip-the-switch-to-activate.) We also do daily offsite backups.

I won't get into the exact sequence of events that would have to happen for us to have permanent, total, and irrevocable data loss, because I don't want to tempt fate, but the way more likely "massive disaster" scenario for us would be a natural disaster that took out the datacenter our servers are hosted in, causing us to have to restore from offsite backup; in the absolute worst case scenario of that scenario (disaster happened right before the nightly offsite backups kicked off) we'd have lost about a day of data, and it would take us a few days to bring the site back up from backup. We could definitely have better failsafes for some of those "total disaster" scenarios, but it would be prohibitively expensive; we settle for "pretty good" risk mitigation instead of perfect.

We also keep an "in case of disaster" cash reserve of no less than six months' operating expenses (this has saved our bacon several times before!) and have a healthy line of credit in case of a sudden need to spend even more than that on really short notice.

Post a comment in response:

Anonymous( )Anonymous This account has disabled anonymous posting.
OpenID( )OpenID You can comment on this post while signed in with an account from many other sites, once you have confirmed your email address. Sign in using OpenID.
Account name:
If you don't have an account you can create one now.
HTML doesn't work in the subject.


If you are unable to use this captcha for any reason, please contact us by email at

Notice: This account is set to log the IP addresses of everyone who comments.
Links will be displayed as unclickable URLs to help prevent spam.