Downtime this morning

[mark]

(For some California local definition of 'morning'!)

About 30 minutes ago one of our databases (sb-db03) locked up and stopped serving traffic. This was an active database, so the site quickly stopped when it could no longer serve requests. Alas.

I have failed us over to a backup database and now everything should be working again.

I'm not sure yet what happened to db03, but am currently investigating and will update this post if I come up with a root cause for the problem. Edit: It's back up and doesn't have any visible problems. Disks are fine, data's intact, etc. The graphs and logs show nothing. We'll have to keep an eye on it and see if it manifests further issues.

Sorry for the trouble, please let me know if you still see any problems!