denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-12-01 01:37 pm

(no subject)

We're having some trouble with our payment system today -- our payment processor is telling us that we have an invalid merchant ID for some reason. (They've recently switched owners and are having some difficulty integrating their systems.)

I'll be checking in with them and seeing what's up. Sorry about the inconvenience!

EDIT, 2:21PM EST: Payments should be fixed in half an hour to an hour. (Turns out we had to update some information about our payment processor with our payment gateway.) I'll update again when our payment gateway confirms that things are back to good.

EDIT, 3PM EST: Our payment gateway has confirmed that everything's back to normal now. Thanks for your patience, all! I'm sorry about the hassle.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark2014-10-14 05:38 pm

Upgrading against the POODLE vulnerability

Hi all,

Today another SSL vulnerability was announced. This one is named POODLE and is, while serious, much less serious than the Heartbleed event from some months ago.

Unfortunately, the only real way to fix the problem is to disable something called "SSLv3" entirely. Basically, this means that we instruct our servers that they are no longer allowed to speak version 3 of the SSL protocol (you can think of it as a language -- we ban this language from our servers). It turns out this is generally OK since most browsers don't actually speak using SSLv3 these days -- you actually use what's called TLS, which is a more modern, better way of protecting the stuff you send across the Internet.

The SSLv3 protocol is actually around 15 years old at this point, and TLS has been out so long that nearly every browser out there supports it. However, shutting off SSLv3 does mean that very old browsers -- IE6, for one -- can no longer talk to Dreamwidth using encryption. In this case, since the encryption wouldn't actually mean anything, we think it's better to not even pretend that it works.

I will be making this change sometime in the next hour or three. This really should impact almost none of you, but there might be one or two and, in that case, I'm sorry. We think it's better to do this so you know you're not actually secure than to let Dreamwidth pretend to be secure.

Edit: This has been deployed. SSLv3 is disabled on Dreamwidth.

Comments and questions welcome, as always!

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark2014-09-04 02:14 pm

DNS change today

Hi all,

We had a brief outage this morning. The cause was an (unexpected) policy change by our DNS provider, Dyn, deciding to shut us off. They had to roll back the change for unrelated reasons so we were back online, but it does mean that we need to migrate off of their service.

ETA: The policy change was that, for about 10 years now -- as long as I've been using Dyn! -- they had no usage/quota limits on their DNS service. Given that DNS requests are tiny and easy to serve, this made sense. They made a business decision recently to establish some (rather tiny) quotas. We're ... quite in excess of them (by some 15,000%) and we don't want to pay in excess of $500 USD/month for DNS service. Amazon's price is 10% of that. They probably tried to contact us, but I don't recall seeing any emails. Anyway, that's it; it's nothing particularly nefarious.

We will be moving our DNS service to Amazon's Route53 service. This kind of migration is fairly easy technically, but if there are problems it will probably mean Dreamwidth will be offline until they can be resolved. And, given the nature of how DNS works, it means that any outage will probably be measured in hours rather than minutes.

I've done my best to ensure that the changeover will go smoothly. If anything happens, though, we'll be on our [twitter.com profile] dreamwidth account to keep everybody apprised of the progress.

The switch will be flipped around 3:30pm PDT / 2230 UTC today, this is in about 90 minutes.

denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-08-12 12:20 pm

(no subject)

Some people are reporting having trouble reaching the site, especially from the UK/EU. Unfortunately, after looking into it, it seems to be an internet routing problem and not something we can fix on our end. Hang tight; our hosting service is pretty good at working around those sorts of problems pretty quickly.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-08-10 11:53 pm

(no subject)

We'll be starting the code push relatively soon; there may be a few blips here and there as we work. I'll update this post when it's all finished!

EDIT: Code has been pushed. Let us know if you encounter any problems! A list of the (many) bugfixes included in this push will be forthcoming.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-08-06 02:21 pm

Code push this weekend

We will be doing a code push this weekend: 9PM PDT Sunday 10 Aug/midnight EDT/5AM GMT Monday 11 Aug/noon PHT. (See in your time zone.)

This push will almost entirely consist of a (very large!) number of fixes for the mobile-friendly styles project, and should fix most of the remaining outstanding issues people have reported.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-07-14 12:23 am

(no subject)

Code push is complete and we're monitoring for any issues. If you spot something that looks off, let us know! The changes covered in this push are in this code tour.

(The site may be a bit sluggish for the next 20 minutes or so while the caches warm back up -- you don't have to tell us about that!)

Notification delays:

An update was posted to [site community profile] dw_news slightly before 0830 EST (see in your time zone). Comment notifications may be delayed for up to an hour or two, due to the high volume of notifications generated by each news post. Please don't worry about missing notifications until at least 1030 EST.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-07-13 11:50 pm

(no subject)

Our code push will begin in approximately 15 minutes. We'll let you know when it's complete!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-07-05 08:31 am

Code push next weekend

We will be doing a code push next weekend: 9PM PDT Sunday 13 Jul/midnight EDT/5AM GMT Monday 14 Jul/noon PHT. (See in your time zone.) This one is mostly bugfixes and small tweaks and shouldn't be much risk for things going kerblooey, and I really hope saying that didn't just jinx us.

(Getting one more code push in before I'm out who knows how long for surgical recovery!)
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-06-10 12:47 am

Code push underway

Tonight's code push is underway! We'll update this entry when we're done.

EDIT: And we're done! As always, we're watching for issues, but let us know of any problems.

Known issues:

* A [site community profile] dw_news post was posted just before 0800 EDT (see in your time zone). Comment notifications may be delayed for up to an hour or two, due to the high volume of notifications generated by each news post. Please don't worry about missing notifications until at least 1000 EDT.

* If you have a custom journal style, and you're seeing entries in the site default (Practicality/Neutral Good) rather than your custom layout, please recompile your layout layer and then try again. (This code push included S2 changes; we recompiled all the site styles, but we can't force custom styles to recompile.)
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-06-08 10:00 pm

Code push tomorrow night

We will be doing a code push tomorrow night, beginning between 9PM and 10PM PDT (midnight-1AM EDT, 5am-6am GMT, View in your time zone.) We'll announce when we're beginning the part that's visible to y'all.

The biggest change in this push is a new and improved frontend for community administration tasks, but there's a bunch more! We'll update you shortly after the push with what new stuff you can expect.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-04-08 08:16 pm

"Heartbleed" security vulnerability

For those who have seen reference today in the press to the "Heartbleed" security vulnerability in OpenSSL, we'd like to reassure you that although we (like a large portion of the internet) were running the affected software, we patched our servers last night and were no longer vulnerable from that point.

We have no reason to believe that anyone was exploiting this vulnerability against us or that any user data has been compromised. We'll be changing our security certificates for extra confidence.

On the other hand, the nature of this vulnerablity means that it's impossible for a website to know for absolute certain whether someone was exploiting it. If someone was exploiting the vulnerability, against us or against any other website, they potentially have access to any information you sent to the site, including your username/password for the site and any data you sent to the site under HTTPS. It's a good idea to change your passwords pretty much everywhere, but don't do it until you can verify that a site is no longer vulnerable.

If you have any questions, feel free to ask!
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark2014-01-26 04:06 pm

Database maintenance

Hi all!

Today we started getting some alerts on the database, so I'm going to do some maintenance to verify the health and wellness of the machine and get it back to a non-alerting status.

I should be able to do this without any downtime, but just in case, you might want to make sure to use your favorite text editor to save a copy of any long entries or comments you're working on.

Once I've got things sorted out, I'll update this with more details for the technically curious.

Update [4:50PM PST]: sb-db06 (the slave) has been rebooted and is recovering, I'm doing system updates on it since the problem looks like a kernel bug (it struck both databases at the same time). Next: master failover then recover the other database.

Update [5:05PM PST]: I'm doing what we call a "master failover" now. This means I'm shifting all traffic from the database that was active (sb-db05) to the spare database (sb-db06). I have to shut off "extra" services like imports, feeds, and searches while this happens.

Update [5:30PM PST]: Well, that was unexpectedly bumpy. Sorry for that. There should be no further bumping, as we're now on the spare database so I can take maintenance on the original master.

Update [6:20PM PST]: If you had userpics not loading, they should be back to normal.

alierak: (Default)
[personal profile] alierak2014-01-26 11:45 am

Database restart

Our database servers have been generating some alerts today, where the monitoring system can't log into the servers to check on them. I can't log in either, but as best I can tell the databases are still running, or you wouldn't be seeing this. There is probably an issue with memory usage or excessive disk I/O, but it's kind of hard to troubleshoot at the moment. I would expect DW to be down for a while at some point today in order to restart the databases and/or reboot the servers. More info when [staff profile] mark's had a chance to look.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2013-11-24 11:59 pm

(no subject)

Code push starting now! We'll update this entry when we're finished.

EDIT: And we're done! We're watching for issues now, but if you spot anything, sing out.

We're working on trying to diagnose and fix the issue of missing notifications. This bug's been fixed. If you were affected, you won't get notifications that you missed getting, but you will get notifications from here on out.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2013-11-23 04:15 am

code push

There will be a code push at midnight EST Sunday 24 Nov (9PM PST Sun 24 Nov, 5AM GMT Monday 25 Nov, see in your time zone)

This push contains some sweeping backend changes, so you either won't notice anything at all, or things will be Very Broken. :) (We're pretty sure things won't be Very Broken, since things have been working out fine in testing, but there's always the chance of things getting screwy when the new changes get widespread adoption.) We'll have everyone on hand to mke sure problems get dealt with quickly.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark2013-10-23 08:56 pm

Importer update

Hi all --

Some have noticed that re-importing to get new comments hasn't been working for a while. This has been fixed; it was an operational issue (the importer cache wasn't being cleaned).

Anyway, if you have been having trouble getting recent comments to import onto DW, things should be working now. Please give it a shot.

Edited: Also, if you are still having importer troubles, please open a new request and let us know here:

http://www.dreamwidth.org/support/submit

Thanks and sorry for the trouble!

denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2013-09-05 03:26 pm

Payment system problems

We're still having trouble tracking down a minor bug with the payment system:

* In some rare cases, trying to complete a payment will result in an error message. It seems to be even odds whether the failure happens before or after your card has been charged, but either way, the items won't be applied to your account. If this happens to you, the error message asks you to open a support request in the Account Payments category: please do so! I'll be able to check whether your card has been charged, and if so, make sure you get the items you paid for.

(EDIT: There was another point here about a different problem that had cropped up since last night, but further investigation turned up that it was only a variant of the above, and only a single payment was affected by it. So, false alarm there!)

I'm really sorry about the hassle, folks! We've been trying to work out what's causing this to happen, but no luck so far. If you ever have questions about whether or not your payment has gone through, just open a support request to ask, and I'll get to it as soon as possible. (This weekend there might be some delay, since my sister's getting married! But usually it's a pretty quick turnaround.)
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2013-08-02 09:51 pm

a few quick notes on ongoing, intermittent issues

We've had a few intermittent issues lately that people have been running into:

1) Random "403 Forbidden" errors when loading site pages, mostly in journals but sometimes on site pages.

2) Shop carts paid for by credit card where you get an error when trying to check out, then the cart is set to "waiting for payment" status.

3) When crossposting to LiveJournal, the crosspost not going through and error messages in your inbox that read "Failed to connect to http://www.livejournal.com/interface/xmlrpc".

All three of these errors are only happening occasionally.

The first two are problems on our end that we're working to track down -- we've added some extra debugging code that should help us pinpoint the cause, and we'll get it fixed as quickly as possible after that. (Things that only happen occasionally are very hard to diagnose and fix, since you can't always know for sure what the cause is, and you can't usually verify that the fixes you're putting into place actually fix the problem.)

The crossposting problem is, unfortunately, on LJ's end and not on ours: that error means that LJ was unreachable at the time the crosspost attempt was sent. If you get the error, your crosspost attempt will retry up to five times, at increasingly-longer intervals, before sending you a final failure notice in your inbox. Once that happens, you can edit the post and re-check the crosspost box, then save the post, to start it trying again. Whether or not it succeeds depends entirely on whether we can reach LJ at the time it tries. (It doesn't matter whether you can load LJ on your own computer when that happens -- the crosspost attempt is sent from our servers, so our servers have to be able to reach LJ, not your computer.)

If you run into problems with a payment, open a support request in the Account Payments category, and I'll get things fixed up for you as soon as I possibly can. (There's only one of me, though, so it probably won't be an immediate response, unless you happen to catch me while I'm sitting right in front of the computer!)

If you run into either of the other two problems, you can let us know by leaving a comment here, just so we can get a rough sense of how often the problems are coming up. We might not be able to tell you anything more than what's in this post, though!

If you're having an issue that isn't one of these three, open a support request, describing the issue as thoroughly as possible, and somebody will help you troubleshoot it.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark2013-07-26 11:28 pm

Code push complete.

Hi all! The code push is downdone.

As always, please let us know if you find anything awry and we'll get right on it!