Dreamwidth Maintenance

Downtime this morning

2013-05-18T14:58:35Z

Posted by: mark

(For some California local definition of 'morning'!)

About 30 minutes ago one of our databases (sb-db03) locked up and stopped serving traffic. This was an active database, so the site quickly stopped when it could no longer serve requests. Alas.

I have failed us over to a backup database and now everything should be working again.

I'm not sure yet what happened to db03, but am currently investigating and will update this post if I come up with a root cause for the problem. Edit: It's back up and doesn't have any visible problems. Disks are fine, data's intact, etc. The graphs and logs show nothing. We'll have to keep an eye on it and see if it manifests further issues.

Sorry for the trouble, please let me know if you still see any problems!

comments

Code push happening

2013-04-23T04:14:21Z

Posted by: mark

Please comment and let us know of anything broken! It'll be live shortly.

comments

Code push in ~90 minutes

2013-04-23T02:24:00Z

Posted by: mark

FYI: We'll be doing the code push in about 90 minutes. I'll post again when it happens, and you can always watch our dreamwidth account on Twitter for updates.

comments

Heads up: Replying from comment notification emails will be changing with next code push

2013-04-20T16:30:49Z

Posted by: denise

Monday night/Tuesday morning's code push will include a change to comment email notifications designed to make it much (much!) easier to reply to comments from the email notification. I wanted to give everyone a heads up ahead of time so y'all don't worry when you see the change!

( Why the change is necessary )

So fu has completely overhauled the process of replying by email. After our next code push, the reply form will be removed from notification emails entirely. Instead, you'll be able to reply to a notification email inside your mail client, just like you'd reply to any other email, and your reply will be posted to the site.

You can read the FAQ -- How do I reply to comments from email notifications? -- before the feature goes live, but ( some questions I anticipate )

Again, this will not be live until the next code push, but if you've got any other questions in advance, let me know! I think that as soon as you see it, you'll fall in love with it as much as I have -- especially if you're trying to juggle multiple accounts. :)

comments

Code push: Monday, April 22nd, 2013

2013-04-19T17:39:16Z

Posted by: mark

Hi all.

We will be doing a codepush in a few days, at about 9PM Pacific on Monday, April 22nd, 2013. Also known as 0400 UTC the 23rd, and a far different time if you're somewhere near fu. :-)

As always, we'll post to our Twitter account and this journal before the push and after to let you know what's going on.

comments

Code push soon

2013-03-02T05:03:56Z

Posted by: mark

Hi all,

We will be doing a code push soon, in about 15 minutes. Please stay tuned!

Edit: Having some technical difficulties with my git setup. Please hold!

EDITED AGAIN: Code is pushed! Let us know if you see anything amiss!

comments

Code push in ~24 hours

2013-03-01T01:09:01Z

Posted by: mark

Hi everybody!

We'll be doing a code push tomorrow about 9PM PST on Friday, March 1st, 2013. This is 0500 UTC on Saturday, March 2nd, 2013.

I don't expect any troubles with this push. As always, please watch our Twitter account and this community for updates! Thanks!

comments

Search changes

2013-02-17T20:48:47Z

Posted by: mark

Hi all,

I just rolled out some search changes. The two main things you should notice: search indexing is much quicker now (it updates with new content every 15 minutes) and also now comments (on paid accounts) are indexed.

Thank you for all of your patience with search. We outgrew our old trusty search machine and it's now retired. The new one is much larger, has faster disks, and should hold us in good standing for a while as we grow.

I'll be around all day and watching this post. Please let me know if you see any problems!

comments

Network service restored

2013-02-12T09:45:58Z

Posted by: denise

At approximately 1AM EST / 6AM GMT, connectivity to the site was lost. It was entirely connectivity-based: our service provider had a networking issue with their own providers. So, our servers were just fine, they just couldn't talk to anybody!

Connectivity appears to have been restored at this point. We're really sorry for the inconvenience.

comments

EDIT: Complete!

2013-01-15T04:51:17Z

Posted by: denise

The site will be going down for maintenance in about ten minutes (at the time I'm writing this). Again, we're going to try for read-only mode rather than "maintenance mode", but that might not be possible.

We'll keep you updated if it looks like we're going to run long, but we're hoping to complete this in 2 hours or less. Watch the offsite status Twitter account for updates.

EDIT: And, we're back! Just goes to show, when mark estimates an hour and you schedule 2 hours just to be safe, everything will finish much faster than that.

We've noticed a few instances of icons not loading, and are looking into it.

comments

Maintenance window in a few hours

2013-01-14T20:43:34Z

Posted by: denise

After yesterday's unexpected outage, we'll need some time to finish the cleanup and get the site back to running in normal (paired database) mode. It's technically possible to do this with the site still up, mark assures me, but it's more futureproof to do it while the site's in maintenance mode. So, we're going to do that tonight.

The maintenance window will begin tonight at midnight EST/9PM PST/5AM UTC. (See this in your timezone.) We're tentatively allocating 2 hours for the work; it may be longer or shorter depending on what we find when we dig into it.

We're going to try doing this with the site in "read-only" mode, where you can read existing entries and comments but not make new ones, rather than maintenance mode, where the entire site is unavailable with a "down for maintenance" splash page replacing all pages. It may prove impossible, though, so if the site switches from read-only mode to down-for-maintenance mode, don't be surprised.

Watch our offsite status for updates -- we'll keep you posted there as things develop.

comments

Site outage over

2013-01-13T22:51:27Z

Posted by: mark

Hi all,

The site outage is over. My apologies for the downtime.

One of our databases filled up its disk and went offline, and this caused the site to stop responding. We failed over to the backup database and everything is now back up and running.

Everything should be working. Please let us know if you see any trouble.

We will need to schedule a maintenance window soon to handle the full database and rebuild the cluster so we have a pair again. Stay tuned to this account to watch for announcements about that.

( More technical detail... )

comments

Code Push Window

2013-01-08T04:25:21Z

Posted by: misskat

The code push scheduled for today will be starting shortly. There is little-to-no expected downtime. The patches being pushed this time around are on this code tour.

A note: any users who have enabled journal-styled icon pages will have that feature turned off as a result of one of the bug fixes. You can turn it back on right away, though! It's under Journal Icons Style on the Display tab in Settings.

-Kat

EDIT: All done! Report any bugs/errors/problems/squids in comments, please. Thanks!

EDIT: Some folks who are using certain Dreamwidth-native styles may have some odd display issues with the custom-styled icon pages. It's a bug! A fix is in the works, and will be live as soon as it's been tested. Should be fixed now! Comment if you continue to have trouble.

comments

Code Push

2013-01-04T09:49:53Z

Posted by: denise

We are tentatively planning a code push on the evening of Monday 7 Jan 11PM EST/8PM PST aka Tue 8 Jan 4AM UTC. (See this in your time zone.)

This should be a low-impact push with minimal, if any, downtime.

comments

Crossposting to LiveJournal, redux

2012-12-31T21:04:42Z

Posted by: denise

Crossposting to LiveJournal continues to be hit-or-miss: if LiveJournal is having technical difficulties or connection problems at the time you try to crosspost, the crosspost will fail and you'll get a long and messy error in your DW inbox.

(To complicate things, sometimes the crosspost attempt will actually succeed, but LJ will report a failure anyway. You should double-check to see if the post actually did crosspost anyway if you get an error.)

Posts will attempt to crosspost several times before failing permanently. Once you get the notification of the permanent failure, you can try to crosspost again by editing the entry and checking the crosspost checkbox.

Also, a very happy new year (Gregorian) from us all to you!

comments

Crossposting to LiveJournal

2012-12-21T09:59:58Z

Posted by: mark

Hi all,

Due to some recent availability problems with accessing the LiveJournal service, crossposts may be slow. The system does automatically retry, but if it's a slow spot it could be minutes or even an hour or two before we can get the entry posted.

Thank you for your patience!

comments

OpenID logins/commenting on the two new webservers

2012-12-09T04:22:14Z

Posted by: denise

After some investigation, we've realized that our two new webservers are having a problem with the OpenID modules installed on them. (For those who care about that sort of thing: it's a case of version incompatability; newer versions of the OpenID modules require (without telling us that they require, ugh) a more up-to-date version of the JSON modules than we have installed or have tested against. So, we have to do some regression testing before we can just upgrade.)

So, for at least the next few days, you might notice that the option for commenting as an OpenID account doesn't appear sometimes when you load a comment form, or if you're logged into an OpenID account you might randomly get weird errors or seem to be logged out, etc. If that happens, just refresh the page until the problem clears up. (When you ask for a page, you get sent to the webserver that's next up, so about 1/4th of the time you'll get one of the webservers with the problem.)

We'll have this fixed as soon as possible! Sorry about the trouble.

comments

the amazing bouncing site search!

2012-12-03T22:44:32Z

Posted by: denise

Alas, the temporary fixes alierak was able to put in place to baby site search along for a little while have proven to not be a match for your text-generating capacity. Search is down again, and probably will be for at least a little while until we're able to achieve a more long-term fix! Sorry about the inconvenience, folks; we'll have it back as soon as we can.

(EDIT: And, back up it goooooooes! Bouncey bouncey bouncey. alierak has been exerting herculean efforts to squeeze every last bit out of the current setup, so search may or may not be available at any given moment. It's Schrödinger's search! So, if you want to search for something, give it a try, but don't be surprised if it's slow, out of date, or down entirely at any given moment.)

comments

Code push coming shortly

2012-12-02T05:01:01Z

Posted by: mark

Hi all!

Edit: This is done, the code has been pushed!

In the next few minutes, we'll be doing a code push on Dreamwidth. You can expect a bit of a rocky experience tonight, since we will be not only deploying new code, but also doubling our caches -- which means that the site will slow down for a while, then will get faster than it was!

If you see any problems, please comment here and let us know. We have all hands on deck to make sure everything goes smoothly.

(PS, special shout out to denise, fu, exor674, and geekosaur who are in IRC helping out or keeping company while this happens. You folks are awesome!)

comments

Updates!

2012-11-30T03:31:47Z

Posted by: denise

1) To follow up on the issue with the site search index not updating: We managed to make a few changes that will allow us to baby things along for a little while longer. (And by 'we', in this case I actually mean alierak!) So, it will take slightly longer for new posts to appear in the search index, but they should be searchable once more.

2) In addition to the code push we have scheduled this weekend (see previous announcement for details and times), we will be adding two new webservers sometime in the next week or so. (And by 'we', I definitely mean mark this time.) Until our provider can build them and Mark can configure them properly and add them into the webserver pool, the site may be a bit sluggish at peak times, usually early evening.

I like this thing we have going on here where all of our problems this week are a result of people using the site to make and do awesome stuff. CARRY ON, FOLKS.

comments

Search indexing not currently updating

2012-11-29T09:04:33Z

Posted by: denise

After some investigation, mark has realized that our little website has grown up to the point where our site search setup is no longer sufficient for the amount of data y'all have generated. This is an exciting problem to have, but it also means that new posts and newly-imported posts won't be indexed into the search database until we can fix it! (And by 'we', I mean 'Mark'.)

We've got a plan that involves a new search database server (to go along with the two new webservers we ordered today), a new database schema that will make indexing, storing, and accessing search data much faster, and a method of storing and retrieving search data that makes more sense for the way usage has developed now that we have more information on what actual usage looks like. (And by 'we', I once again mean 'Mark'. There is a theme here.)

The plan is going to take a week or two to implement, though, and until then, search results will continue to be out of date. Anything that is already in the search database (as of about 10 days ago) will still be searchable, but newly-made posts and newly-imported posts won't show up until after we can fix it.

We'll let you know when the problem is fixed. Until then, continue on being awesome and making stuff: we'll catch up to you as soon as possible.

comments

Code push this weekend

2012-11-29T00:45:47Z

Posted by: denise

We are tentatively planning a code push this weekend, at roughly midnight Saturday EST/9PM Saturday PST/5AM Sunday UTC. (See this in your time zone.)

With luck you won't even notice it (except for the new stuff afterwards, of course!) as it shouldn't involve downtime, but we'll keep you posted!

comments

Imported content removal tool

2012-11-16T02:13:25Z

Posted by: denise

For people who got bitten by LJ's bug that was causing duplicate content in your account after importing content from LJ, we've finished the tool that will help you clean up your account without having to delete each individual post or comment.

This tool will remove all imported content from your account, not just duplicate imported content. (Basically, there's no way for us to tell which version of a post or comment is the 'original' and which is the duplicate, so we need to remove them both and let you re-import from scratch.) Once you've removed all imported content, you can then re-import your content from scratch.

Warnings: This can't be undone. Once you hit the button to delete all imported content, it can't be stopped. It will remove all imported content from your account, no matter how long ago you imported it. If you've deleted posts or comments on LJ since you first imported, they can't be re-imported into your account on Dreamwidth after you remove all imported content.

Reassurances: It will only touch imported content (with the exception of: If an entry was imported, and then someone commented on that entry on Dreamwidth, running this tool will delete that entry and all associated comments, including any comments that were made on Dreamwidth.) We've tested it pretty heavily (although there's always a chance for weird bugs).

If you want to use the tool to clean up your account, it's here.

EDIT: Also, I forgot to mention: if you're one of the people who were affected by the issue with the system telling you that your comments had imported successfully but the comments weren't appearing in your account, you can now try your import again, and it should work this time!

comments

Update on the importer

2012-11-03T00:59:15Z

Posted by: mark

(I am posting this on behalf of zarhooie, one of our Support Administrators.)

Hello Dreamwidth! Kat here!

As you may have noticed, we are experiencing some issues with importing comments from LiveJournal. They made some changes to their code, which broke the way our importer retrieves comments. Mark knows what is wrong, and is working on it. Unfortunately, it's going to be a day or two before it's fixed.

For right now, hold tight and don't retry your import. Once it's fixed, someone will post here and then you can try again. Thank you for your patience!

Best,
Kat (zarhooie)
Dreamwidth Support

comments

images and JS loading problems

2012-10-21T12:18:03Z

Posted by: denise

Some people are having problems loading images (icons in particular) and static content such as Javascript right now -- we've been testing out a new content delivery network (CDN) to speed up access to static content, and it seems like a small percentage of people can't load things from the network. Hang tight, we'll get it fixed as soon as possible.

comments