denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
We've been hacking away in person at the conference we went to this week, and we'd like to share the fruit of our labors with you all! There'll be a code push tonight (6/7) at 9PM CDT, which is 10PM EDT/7PM PDT/2AM GMT (6/8). (Convert to your time zone!)

We don't consider this one "high risk", so (*knocks wood*) it should be pretty uneventful.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi all!

The new database machines I ordered are now installed and spinning up. They're in the beginning phases of their life, which means I've moved a few test accounts (a few communities and some other random people) and will be watching how they behave over the next day or two to make sure that everything is happy.

The new database cluster has been christened Epsilon Eridani and will soon be the home for all of our users.

You should really not expect to see anything yet, but take this post as fore-warning that sometime soon (I'll post again) I will start moving accounts in earnest. You can expect brief bouts of "read-only mode" when this happens, so if you see that starting to pop up around the site in the next few days -- that's why!

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

(For some California local definition of 'morning'!)

About 30 minutes ago one of our databases (sb-db03) locked up and stopped serving traffic. This was an active database, so the site quickly stopped when it could no longer serve requests. Alas.

I have failed us over to a backup database and now everything should be working again.

I'm not sure yet what happened to db03, but am currently investigating and will update this post if I come up with a root cause for the problem. Edit: It's back up and doesn't have any visible problems. Disks are fine, data's intact, etc. The graphs and logs show nothing. We'll have to keep an eye on it and see if it manifests further issues.

Sorry for the trouble, please let me know if you still see any problems!

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
Please comment and let us know of anything broken! It'll be live shortly.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
FYI: We'll be doing the code push in about 90 minutes. I'll post again when it happens, and you can always watch our [twitter.com profile] dreamwidth account on Twitter for updates.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
Monday night/Tuesday morning's code push will include a change to comment email notifications designed to make it much (much!) easier to reply to comments from the email notification. I wanted to give everyone a heads up ahead of time so y'all don't worry when you see the change!

Why the change is necessary )

So [staff profile] fu has completely overhauled the process of replying by email. After our next code push, the reply form will be removed from notification emails entirely. Instead, you'll be able to reply to a notification email inside your mail client, just like you'd reply to any other email, and your reply will be posted to the site.

You can read the FAQ -- How do I reply to comments from email notifications? -- before the feature goes live, but some questions I anticipate )

Again, this will not be live until the next code push, but if you've got any other questions in advance, let me know! I think that as soon as you see it, you'll fall in love with it as much as I have -- especially if you're trying to juggle multiple accounts. :)
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi all.

We will be doing a codepush in a few days, at about 9PM Pacific on Monday, April 22nd, 2013. Also known as 0400 UTC the 23rd, and a far different time if you're somewhere near [staff profile] fu. :-)

As always, we'll post to our Twitter account and this journal before the push and after to let you know what's going on.

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi all,

We will be doing a code push soon, in about 15 minutes. Please stay tuned!

Edit: Having some technical difficulties with my git setup. Please hold!

EDITED AGAIN: Code is pushed! Let us know if you see anything amiss!

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi everybody!

We'll be doing a code push tomorrow about 9PM PST on Friday, March 1st, 2013. This is 0500 UTC on Saturday, March 2nd, 2013.

I don't expect any troubles with this push. As always, please watch our Twitter account and this community for updates! Thanks!

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
Hi all,

I just rolled out some search changes. The two main things you should notice: search indexing is much quicker now (it updates with new content every 15 minutes) and also now comments (on paid accounts) are indexed.

Thank you for all of your patience with search. We outgrew our old trusty search machine and it's now retired. The new one is much larger, has faster disks, and should hold us in good standing for a while as we grow.

I'll be around all day and watching this post. Please let me know if you see any problems!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
At approximately 1AM EST / 6AM GMT, connectivity to the site was lost. It was entirely connectivity-based: our service provider had a networking issue with their own providers. So, our servers were just fine, they just couldn't talk to anybody!

Connectivity appears to have been restored at this point. We're really sorry for the inconvenience.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
The site will be going down for maintenance in about ten minutes (at the time I'm writing this). Again, we're going to try for read-only mode rather than "maintenance mode", but that might not be possible.

We'll keep you updated if it looks like we're going to run long, but we're hoping to complete this in 2 hours or less. Watch the offsite status Twitter account for updates.

EDIT: And, we're back! Just goes to show, when [staff profile] mark estimates an hour and you schedule 2 hours just to be safe, everything will finish much faster than that.

We've noticed a few instances of icons not loading, and are looking into it.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
After yesterday's unexpected outage, we'll need some time to finish the cleanup and get the site back to running in normal (paired database) mode. It's technically possible to do this with the site still up, [staff profile] mark assures me, but it's more futureproof to do it while the site's in maintenance mode. So, we're going to do that tonight.

The maintenance window will begin tonight at midnight EST/9PM PST/5AM UTC. (See this in your timezone.) We're tentatively allocating 2 hours for the work; it may be longer or shorter depending on what we find when we dig into it.

We're going to try doing this with the site in "read-only" mode, where you can read existing entries and comments but not make new ones, rather than maintenance mode, where the entire site is unavailable with a "down for maintenance" splash page replacing all pages. It may prove impossible, though, so if the site switches from read-only mode to down-for-maintenance mode, don't be surprised.

Watch our offsite status for updates -- we'll keep you posted there as things develop.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
Hi all,

The site outage is over. My apologies for the downtime.

One of our databases filled up its disk and went offline, and this caused the site to stop responding. We failed over to the backup database and everything is now back up and running.

Everything should be working. Please let us know if you see any trouble.

We will need to schedule a maintenance window soon to handle the full database and rebuild the cluster so we have a pair again. Stay tuned to this account to watch for announcements about that.

More technical detail... )
misskat: Dreamsheep in the snow, with MissKat on top and Minion #2 underneath (Dreamwidth Minion #2)
[staff profile] misskat
The code push scheduled for today will be starting shortly. There is little-to-no expected downtime. The patches being pushed this time around are on this code tour.

A note: any users who have enabled journal-styled icon pages will have that feature turned off as a result of one of the bug fixes. You can turn it back on right away, though! It's under Journal Icons Style on the Display tab in Settings.

-Kat

EDIT: All done! Report any bugs/errors/problems/squids in comments, please. Thanks!

EDIT: Some folks who are using certain Dreamwidth-native styles may have some odd display issues with the custom-styled icon pages. It's a bug! A fix is in the works, and will be live as soon as it's been tested. Should be fixed now! Comment if you continue to have trouble.

Code Push

Jan. 4th, 2013 04:45 am
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
We are tentatively planning a code push on the evening of Monday 7 Jan 11PM EST/8PM PST aka Tue 8 Jan 4AM UTC. (See this in your time zone.)

This should be a low-impact push with minimal, if any, downtime.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
Crossposting to LiveJournal continues to be hit-or-miss: if LiveJournal is having technical difficulties or connection problems at the time you try to crosspost, the crosspost will fail and you'll get a long and messy error in your DW inbox.

(To complicate things, sometimes the crosspost attempt will actually succeed, but LJ will report a failure anyway. You should double-check to see if the post actually did crosspost anyway if you get an error.)

Posts will attempt to crosspost several times before failing permanently. Once you get the notification of the permanent failure, you can try to crosspost again by editing the entry and checking the crosspost checkbox.

Also, a very happy new year (Gregorian) from us all to you!
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
Hi all,

Due to some recent availability problems with accessing the LiveJournal service, crossposts may be slow. The system does automatically retry, but if it's a slow spot it could be minutes or even an hour or two before we can get the entry posted.

Thank you for your patience!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
After some investigation, we've realized that our two new webservers are having a problem with the OpenID modules installed on them. (For those who care about that sort of thing: it's a case of version incompatability; newer versions of the OpenID modules require (without telling us that they require, ugh) a more up-to-date version of the JSON modules than we have installed or have tested against. So, we have to do some regression testing before we can just upgrade.)

So, for at least the next few days, you might notice that the option for commenting as an OpenID account doesn't appear sometimes when you load a comment form, or if you're logged into an OpenID account you might randomly get weird errors or seem to be logged out, etc. If that happens, just refresh the page until the problem clears up. (When you ask for a page, you get sent to the webserver that's next up, so about 1/4th of the time you'll get one of the webservers with the problem.)

We'll have this fixed as soon as possible! Sorry about the trouble.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
Alas, the temporary fixes [personal profile] alierak was able to put in place to baby site search along for a little while have proven to not be a match for your text-generating capacity. Search is down again, and probably will be for at least a little while until we're able to achieve a more long-term fix! Sorry about the inconvenience, folks; we'll have it back as soon as we can.

(EDIT: And, back up it goooooooes! Bouncey bouncey bouncey. [personal profile] alierak has been exerting herculean efforts to squeeze every last bit out of the current setup, so search may or may not be available at any given moment. It's Schrödinger's search! So, if you want to search for something, give it a try, but don't be surprised if it's slow, out of date, or down entirely at any given moment.)
Page generated Apr. 18th, 2014 12:38 am
Powered by Dreamwidth Studios