This code push contains some rewrites/conversions of various pages on the site, so things will look a little different than what you're used to. The most obvious change will probably be to the Create Entry page -- it's not a redesign, and things will continue to behave the exact same way they have been, they'll just look a little bit different. Do not adjust the horizontal, do not adjust the vertical.
EDIT: Sorry, mark got started before I could update! We are in the middle of pushing now.
We'll update you again right before we're ready to get started.
We've fixed things, and we're contacting our captcha provider to make sure we don't run into this again in the future.
Thanks all for your patience! And lots of thanks to everyone who reported the issue and all our support people who made sure we developers knew something was up. If you notice any further weirdness, please open a support request and we'll look into it.
I'll be checking in with them and seeing what's up. Sorry about the inconvenience!
EDIT, 2:21PM EST: Payments should be fixed in half an hour to an hour. (Turns out we had to update some information about our payment processor with our payment gateway.) I'll update again when our payment gateway confirms that things are back to good.
EDIT, 3PM EST: Our payment gateway has confirmed that everything's back to normal now. Thanks for your patience, all! I'm sorry about the hassle.
Today another SSL vulnerability was announced. This one is named POODLE and is, while serious, much less serious than the Heartbleed event from some months ago.
Unfortunately, the only real way to fix the problem is to disable something called "SSLv3" entirely. Basically, this means that we instruct our servers that they are no longer allowed to speak version 3 of the SSL protocol (you can think of it as a language -- we ban this language from our servers). It turns out this is generally OK since most browsers don't actually speak using SSLv3 these days -- you actually use what's called TLS, which is a more modern, better way of protecting the stuff you send across the Internet.
The SSLv3 protocol is actually around 15 years old at this point, and TLS has been out so long that nearly every browser out there supports it. However, shutting off SSLv3 does mean that very old browsers -- IE6, for one -- can no longer talk to Dreamwidth using encryption. In this case, since the encryption wouldn't actually mean anything, we think it's better to not even pretend that it works.
I will be making this change sometime in the next hour or three. This really should impact almost none of you, but there might be one or two and, in that case, I'm sorry. We think it's better to do this so you know you're not actually secure than to let Dreamwidth pretend to be secure.
Edit: This has been deployed. SSLv3 is disabled on Dreamwidth.
Comments and questions welcome, as always!
We had a brief outage this morning. The cause was an (unexpected) policy change by our DNS provider, Dyn, deciding to shut us off. They had to roll back the change for unrelated reasons so we were back online, but it does mean that we need to migrate off of their service.
ETA: The policy change was that, for about 10 years now -- as long as I've been using Dyn! -- they had no usage/quota limits on their DNS service. Given that DNS requests are tiny and easy to serve, this made sense. They made a business decision recently to establish some (rather tiny) quotas. We're ... quite in excess of them (by some 15,000%) and we don't want to pay in excess of $500 USD/month for DNS service. Amazon's price is 10% of that. They probably tried to contact us, but I don't recall seeing any emails. Anyway, that's it; it's nothing particularly nefarious.
We will be moving our DNS service to Amazon's Route53 service. This kind of migration is fairly easy technically, but if there are problems it will probably mean Dreamwidth will be offline until they can be resolved. And, given the nature of how DNS works, it means that any outage will probably be measured in hours rather than minutes.
I've done my best to ensure that the changeover will go smoothly. If anything happens, though, we'll be on our dreamwidth account to keep everybody apprised of the progress.
The switch will be flipped around 3:30pm PDT / 2230 UTC today, this is in about 90 minutes.
EDIT: Code has been pushed. Let us know if you encounter any problems! A list of the (many) bugfixes included in this push will be forthcoming.
This push will almost entirely consist of a (very large!) number of fixes for the mobile-friendly styles project, and should fix most of the remaining outstanding issues people have reported.
(The site may be a bit sluggish for the next 20 minutes or so while the caches warm back up -- you don't have to tell us about that!)
An update was posted to dw_news slightly before 0830 EST (see in your time zone). Comment notifications may be delayed for up to an hour or two, due to the high volume of notifications generated by each news post. Please don't worry about missing notifications until at least 1030 EST.
(Getting one more code push in before I'm out who knows how long for surgical recovery!)
EDIT: And we're done! As always, we're watching for issues, but let us know of any problems.
* A dw_news post was posted just before 0800 EDT (see in your time zone). Comment notifications may be delayed for up to an hour or two, due to the high volume of notifications generated by each news post. Please don't worry about missing notifications until at least 1000 EDT.
* If you have a custom journal style, and you're seeing entries in the site default (Practicality/Neutral Good) rather than your custom layout, please recompile your layout layer and then try again. (This code push included S2 changes; we recompiled all the site styles, but we can't force custom styles to recompile.)
The biggest change in this push is a new and improved frontend for community administration tasks, but there's a bunch more! We'll update you shortly after the push with what new stuff you can expect.
We have no reason to believe that anyone was exploiting this vulnerability against us or that any user data has been compromised. We'll be changing our security certificates for extra confidence.
On the other hand, the nature of this vulnerablity means that it's impossible for a website to know for absolute certain whether someone was exploiting it. If someone was exploiting the vulnerability, against us or against any other website, they potentially have access to any information you sent to the site, including your username/password for the site and any data you sent to the site under HTTPS. It's a good idea to change your passwords pretty much everywhere, but don't do it until you can verify that a site is no longer vulnerable.
If you have any questions, feel free to ask!
Today we started getting some alerts on the database, so I'm going to do some maintenance to verify the health and wellness of the machine and get it back to a non-alerting status.
I should be able to do this without any downtime, but just in case, you might want to make sure to use your favorite text editor to save a copy of any long entries or comments you're working on.
Once I've got things sorted out, I'll update this with more details for the technically curious.
Update [4:50PM PST]:
sb-db06 (the slave) has been rebooted and is recovering, I'm doing system updates on it since the problem looks like a kernel bug (it struck both databases at the same time). Next: master failover then recover the other database.
Update [5:05PM PST]: I'm doing what we call a "master failover" now. This means I'm shifting all traffic from the database that was active (
sb-db05) to the spare database (
sb-db06). I have to shut off "extra" services like imports, feeds, and searches while this happens.
Update [5:30PM PST]: Well, that was unexpectedly bumpy. Sorry for that. There should be no further bumping, as we're now on the spare database so I can take maintenance on the original master.
Update [6:20PM PST]: If you had userpics not loading, they should be back to normal.
EDIT: And we're done! We're watching for issues now, but if you spot anything, sing out.
This push contains some sweeping backend changes, so you either won't notice anything at all, or things will be Very Broken. :) (We're pretty sure things won't be Very Broken, since things have been working out fine in testing, but there's always the chance of things getting screwy when the new changes get widespread adoption.) We'll have everyone on hand to mke sure problems get dealt with quickly.