mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
Mark Smith ([staff profile] mark) wrote in [site community profile] dw_maintenance2013-07-01 05:17 pm

Database maintenance

Hi all,

I did some database maintenance today -- moving our workers around! -- and this caused a glitch in the replication between our old databases and the new ones, so the new ones weren't getting all the updated data.

What this means to you: if you saw problems trying to update your access list or subscription filters, or with community invitations, or viewing support requests, that was caused by the glitch in replication. I'm really sorry for the inconvenience.

This particular issue won't recur, since it was caused by a very specific circumstance related to moving the workers around. Since I'm done moving them, the problem won't happen again.


Right now we're migrating from our old master databases (db01 and db02) to the new pair (db05 and db06). To do this sanely, I have it set up in a replication chain so that any changes made at the top will trickle down to the bottom ones, like this:

db01 -> db02 -> db05 -> db06

The idea is that, to migrate seamlessly from the old ones to the new ones, at some point in time I just change the configuration files that used to say 01/02 and make them say 05/06. Then, magically and nearly instantaneously, we're using the new databases and after some days I can get rid of the old ones.

Anyway, today I moved our TheSchwartz based workers (they do notifications, emails, and some other tasks). I switched them to the new database cluster -- but of course, nothing is actually instantaneous. What happened was that some of the web servers started using db05 a split-second (literally) before some of the others, so we had a few hundred milliseconds where db01 (the OLD master) received some writes after db05 (new one) did.

The problem was then that both databases assigned the same number to different jobs. (When a job gets inserted, it gets assigned an ID. Since both databases had a slight overlap where they both thought they were boss, both created the same ID!)

This is where the sadness happened, because when db05 tried to replicate the commands that db01 had done in that split-second, there was a conflict: two jobs had the same ID. So, db05 stopped replicating from db01 (technically db02) and we didn't have an alert on it because it's going to be a master (i.e., it's not supposed to be replicating long term, so I never set up a replication alarm for it).

Anyway, someone reported an issue which I tracked down to a replication problem. It's been fixed, the database is now fully replicated, and the problem won't repeat because the switchover has already happened. db05 is the master for generating IDs for jobs now, db01 is deprecated.

Thanks for reading.

Post a comment in response:

Anonymous( )Anonymous This account has disabled anonymous posting.
OpenID( )OpenID You can comment on this post while signed in with an account from many other sites, once you have confirmed your email address. Sign in using OpenID.
Account name:
If you don't have an account you can create one now.
HTML doesn't work in the subject.


If you are unable to use this captcha for any reason, please contact us by email at

Notice: This account is set to log the IP addresses of everyone who comments.
Links will be displayed as unclickable URLs to help prevent spam.