mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi all,

The code push and downtime will be happening in one hour. Remember, the site will be down for ~10 minutes while I do a database failover to move us completely on to the new hardware.

I will post again when the push happens, and as always, you can check out our [twitter.com profile] dreamwidth account on Twitter to keep up with the code push!

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi all!

I'd like to do a brief site maintenance and code push in about 32 hours. The scheduled time:

  • 2100 PDT Thursday
  • 0400 UTC Friday

This code push will involve a period of downtime. I'm estimating it will take about 10 minutes. During that window, I will be switching us from our old sb-db01 database master to our new sb-db05 cluster. I have to take the site offline for this since it's what we call a master failover, and our system isn't designed to do that without downtime.

After this maintenance, we will be completely on new hardware -- and all of the trusty hardware we've had for the past three years since moving to ServerBeach will be completely retired.

As always, there will be another post here as well as on our Twitter account when the time comes. Feel free to shoot me any questions or comments, I'll watch this post!

PS HELLO PLURKERS.

alierak: (ninja)
[personal profile] alierak
I'm going to restart Varnish on the main loadbalancer at 8am CDT (13:00 UTC). Dreamwidth will be completely unavailable for a very brief time, and then maybe a little slower for a while as the cache gets refilled. But when it's done, some things should be faster than they are now. Since we have new servers with more RAM, it's time to use some of it to cache more user icons.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
We're getting multiple reports of crossposting to LiveJournal not working, with an error message of "Failed to crosspost entry to [username]@LiveJournal: Failed to connect to http://www.livejournal.com/interface/xmlrpc." We're looking into whether this is a problem on LJ's end or our end (and if it's our end, we'll do our best to fix it!)

EDIT: This problem has been fixed now!
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi all,

I did some database maintenance today -- moving our workers around! -- and this caused a glitch in the replication between our old databases and the new ones, so the new ones weren't getting all the updated data.

What this means to you: if you saw problems trying to update your access list or subscription filters, or with community invitations, or viewing support requests, that was caused by the glitch in replication. I'm really sorry for the inconvenience.

This particular issue won't recur, since it was caused by a very specific circumstance related to moving the workers around. Since I'm done moving them, the problem won't happen again.

A more technical explanation follows... )

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
Hi all,

As part of our new hardware project, I'm going to be failing us over to our new load balancers. This will involve a brief downtime for the site while everything fails over, but it should be less than 60 seconds.

Thanks for your patience, and sorry for the interruption!
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
The payment system is back online. It was my fault; I was moving it to our new hardware, but I didn't realize there is a code change that I have to make. (For the details curious, the underlying SSL module we use was upgraded, and it now requires you to add some more options when you use it.)

I have cleared out the pending queue of payments, so that we shouldn't have charged for anything in the past 24 hours, and that should mean there are no doubled (or more) payments. Please, of course, let us know if that's the case though, and we'll take care of it!

Sorry for the trouble!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
The backend system that runs payments is temporarily unavailable, and will be fixed as soon as possible. If you've tried to make a payment at any time between last night & now and gotten an endless wait, your payment is almost certainly in the queue to be processed as soon as the backend is back up & running -- you don't need to submit it again.

If you wind up getting multiple charges when it comes back up (for instance, if you re-submitted the form, thinking that your internet connection was to blame) you can open a support request (in the Account Payments category) after the payment is processed and I'll issue a refund to your card for the extra charges.

We're really sorry about the downtime!

Code pushed

Jun. 7th, 2013 11:31 pm
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
Hi all!

The code has been pushed. As always, please report problems here! We have lots of hands on deck and ready to jump on things that might be awry. Thanks!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
We've been hacking away in person at the conference we went to this week, and we'd like to share the fruit of our labors with you all! There'll be a code push tonight (6/7) at 9PM CDT, which is 10PM EDT/7PM PDT/2AM GMT (6/8). (Convert to your time zone!)

We don't consider this one "high risk", so (*knocks wood*) it should be pretty uneventful.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi all!

The new database machines I ordered are now installed and spinning up. They're in the beginning phases of their life, which means I've moved a few test accounts (a few communities and some other random people) and will be watching how they behave over the next day or two to make sure that everything is happy.

The new database cluster has been christened Epsilon Eridani and will soon be the home for all of our users.

You should really not expect to see anything yet, but take this post as fore-warning that sometime soon (I'll post again) I will start moving accounts in earnest. You can expect brief bouts of "read-only mode" when this happens, so if you see that starting to pop up around the site in the next few days -- that's why!

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

(For some California local definition of 'morning'!)

About 30 minutes ago one of our databases (sb-db03) locked up and stopped serving traffic. This was an active database, so the site quickly stopped when it could no longer serve requests. Alas.

I have failed us over to a backup database and now everything should be working again.

I'm not sure yet what happened to db03, but am currently investigating and will update this post if I come up with a root cause for the problem. Edit: It's back up and doesn't have any visible problems. Disks are fine, data's intact, etc. The graphs and logs show nothing. We'll have to keep an eye on it and see if it manifests further issues.

Sorry for the trouble, please let me know if you still see any problems!

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
Please comment and let us know of anything broken! It'll be live shortly.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
FYI: We'll be doing the code push in about 90 minutes. I'll post again when it happens, and you can always watch our [twitter.com profile] dreamwidth account on Twitter for updates.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
Monday night/Tuesday morning's code push will include a change to comment email notifications designed to make it much (much!) easier to reply to comments from the email notification. I wanted to give everyone a heads up ahead of time so y'all don't worry when you see the change!

Why the change is necessary )

So [staff profile] fu has completely overhauled the process of replying by email. After our next code push, the reply form will be removed from notification emails entirely. Instead, you'll be able to reply to a notification email inside your mail client, just like you'd reply to any other email, and your reply will be posted to the site.

You can read the FAQ -- How do I reply to comments from email notifications? -- before the feature goes live, but some questions I anticipate )

Again, this will not be live until the next code push, but if you've got any other questions in advance, let me know! I think that as soon as you see it, you'll fall in love with it as much as I have -- especially if you're trying to juggle multiple accounts. :)
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi all.

We will be doing a codepush in a few days, at about 9PM Pacific on Monday, April 22nd, 2013. Also known as 0400 UTC the 23rd, and a far different time if you're somewhere near [staff profile] fu. :-)

As always, we'll post to our Twitter account and this journal before the push and after to let you know what's going on.

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi all,

We will be doing a code push soon, in about 15 minutes. Please stay tuned!

Edit: Having some technical difficulties with my git setup. Please hold!

EDITED AGAIN: Code is pushed! Let us know if you see anything amiss!

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi everybody!

We'll be doing a code push tomorrow about 9PM PST on Friday, March 1st, 2013. This is 0500 UTC on Saturday, March 2nd, 2013.

I don't expect any troubles with this push. As always, please watch our Twitter account and this community for updates! Thanks!

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
Hi all,

I just rolled out some search changes. The two main things you should notice: search indexing is much quicker now (it updates with new content every 15 minutes) and also now comments (on paid accounts) are indexed.

Thank you for all of your patience with search. We outgrew our old trusty search machine and it's now retired. The new one is much larger, has faster disks, and should hold us in good standing for a while as we grow.

I'll be around all day and watching this post. Please let me know if you see any problems!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
At approximately 1AM EST / 6AM GMT, connectivity to the site was lost. It was entirely connectivity-based: our service provider had a networking issue with their own providers. So, our servers were just fine, they just couldn't talk to anybody!

Connectivity appears to have been restored at this point. We're really sorry for the inconvenience.
Page generated Sep. 22nd, 2014 10:10 am
Powered by Dreamwidth Studios