I will be doing a small code push tonight in about an hour or two. It hasn't been long since our last push so this one isn't particularly large, hence the short notice.
I'll post on Twitter when we start. There should be no downtime for this.
|You're viewing dw_maintenance|
Create a Dreamwidth Account Learn More
The code push and downtime will be happening in one hour. Remember, the site will be down for ~10 minutes while I do a database failover to move us completely on to the new hardware.
I will post again when the push happens, and as always, you can check out our dreamwidth account on Twitter to keep up with the code push!
I'd like to do a brief site maintenance and code push in about 32 hours. The scheduled time:
This code push will involve a period of downtime. I'm estimating it will take about 10 minutes. During that window, I will be switching us from our old
sb-db01 database master to our new
sb-db05 cluster. I have to take the site offline for this since it's what we call a master failover, and our system isn't designed to do that without downtime.
After this maintenance, we will be completely on new hardware -- and all of the trusty hardware we've had for the past three years since moving to ServerBeach will be completely retired.
As always, there will be another post here as well as on our Twitter account when the time comes. Feel free to shoot me any questions or comments, I'll watch this post!
PS HELLO PLURKERS.
I did some database maintenance today -- moving our workers around! -- and this caused a glitch in the replication between our old databases and the new ones, so the new ones weren't getting all the updated data.
What this means to you: if you saw problems trying to update your access list or subscription filters, or with community invitations, or viewing support requests, that was caused by the glitch in replication. I'm really sorry for the inconvenience.
This particular issue won't recur, since it was caused by a very specific circumstance related to moving the workers around. Since I'm done moving them, the problem won't happen again.
The new database machines I ordered are now installed and spinning up. They're in the beginning phases of their life, which means I've moved a few test accounts (a few communities and some other random people) and will be watching how they behave over the next day or two to make sure that everything is happy.
The new database cluster has been christened Epsilon Eridani and will soon be the home for all of our users.
You should really not expect to see anything yet, but take this post as fore-warning that sometime soon (I'll post again) I will start moving accounts in earnest. You can expect brief bouts of "read-only mode" when this happens, so if you see that starting to pop up around the site in the next few days -- that's why!
(For some California local definition of 'morning'!)
About 30 minutes ago one of our databases (sb-db03) locked up and stopped serving traffic. This was an active database, so the site quickly stopped when it could no longer serve requests. Alas.
I have failed us over to a backup database and now everything should be working again.
I'm not sure yet what happened to db03, but am currently investigating and will update this post if I come up with a root cause for the problem. Edit: It's back up and doesn't have any visible problems. Disks are fine, data's intact, etc. The graphs and logs show nothing. We'll have to keep an eye on it and see if it manifests further issues.
Sorry for the trouble, please let me know if you still see any problems!
We will be doing a codepush in a few days, at about 9PM Pacific on Monday, April 22nd, 2013. Also known as 0400 UTC the 23rd, and a far different time if you're somewhere near fu. :-)
As always, we'll post to our Twitter account and this journal before the push and after to let you know what's going on.
We'll be doing a code push tomorrow about 9PM PST on Friday, March 1st, 2013. This is 0500 UTC on Saturday, March 2nd, 2013.
I don't expect any troubles with this push. As always, please watch our Twitter account and this community for updates! Thanks!