denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2015-02-27 11:30 pm

(no subject)

If you're seeing slow page load times, pages not fully loading, missing icons, 'naked' pages (the text of the page only, without any styling, etc): please shift-refresh your browser, clear your browser cache, and then just hang tight. We're switching CDN providers, so your browser may have cached the wrong copy of things.

If the problem hasn't cleared up by tomorrow, then let us know and we'll look into it further!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2015-02-05 05:46 am

Email issues to Hotmail continue, joined by issues to Yahoo

For the past week and a half or so, Hotmail has been rejecting all email from us. Sadly, Yahoo has now joined them. This means that if your confirmed email address is @hotmail.com or @yahoo.com, you won't get any email from Dreamwidth at all. (This includes comment notification emails, password reset emails, lost username emails, confirmations/receipts for payments, email from your @dreamwidth.org forwarding address ... basically, if it's email and it's from us, it won't reach you.)

This is a block at the system level, so there isn't anything you can do to work around it. It's not due to spam filters on your individual account, so whitelisting Dreamwidth in your mail application won't work to fix things this time: these providers are blocking all mail from us, across the board. When a provider makes that kind of block, they don't discuss the exact reasons that led to the blocking, but our best guess is that it's due to a combination of the amount of email we send out (especially email with highly similar subject lines and content, since -- for instance -- a notification of a new comment to an entry quotes the entry) and an uptick in the amount of spam sent to @dreamwidth.org forwarding email addresses, since that forwarded spam can look like we were the ones to send it.

We've gone through the process to request unblocking from the providers directly, but haven't seen any progress there yet. We are, however, actively working on alternate solutions that will reduce the risk of us being identified as spammers in the future -- we ran the first test yesterday and results are looking promising. We'll keep you posted on the progress.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2015-01-18 03:48 am

Code push finished, fixing immediate bugs now

We've finished pushing the new code and will be working on fixing immediately obvious bugs now!

We're looking into problems with the new Create/Edit entries page not working properly on Chrome.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2015-01-18 08:21 pm

(no subject)

We are (inevitably) running a little bit late -- the code push will begin in about half an hour or so. We'll update this entry when we're beginning.

This code push contains some rewrites/conversions of various pages on the site, so things will look a little different than what you're used to. The most obvious change will probably be to the Create Entry page -- it's not a redesign, and things will continue to behave the exact same way they have been, they'll just look a little bit different. Do not adjust the horizontal, do not adjust the vertical.

EDIT: Sorry, [staff profile] mark got started before I could update! We are in the middle of pushing now.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2015-01-16 06:28 pm

Code push on Saturday/Sunday

We will be doing a code push this weekend! [staff profile] mark, [staff profile] fu, and I are in New Zealand for Linux Conf AU, so the code push is planned for Sunday, 18 January, at 8PM New Zealand time (GMT +13). (See this in your time zone.) This is Sunday 2AM EST/7AM GMT/Saturday 11PM PDT.

We'll update you again right before we're ready to get started.
fu: Close-up of Fu, bringing a scoop of water to her mouth (Default)
[staff profile] fu2015-01-03 08:20 am

Captcha issues

We had some issues with our captcha earlier today which prevented people from making new accounts and prevented some comments from being posted.

We've fixed things, and we're contacting our captcha provider to make sure we don't run into this again in the future.

Thanks all for your patience! And lots of thanks to everyone who reported the issue and all our support people who made sure we developers knew something was up. If you notice any further weirdness, please open a support request and we'll look into it.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-12-01 01:37 pm

(no subject)

We're having some trouble with our payment system today -- our payment processor is telling us that we have an invalid merchant ID for some reason. (They've recently switched owners and are having some difficulty integrating their systems.)

I'll be checking in with them and seeing what's up. Sorry about the inconvenience!

EDIT, 2:21PM EST: Payments should be fixed in half an hour to an hour. (Turns out we had to update some information about our payment processor with our payment gateway.) I'll update again when our payment gateway confirms that things are back to good.

EDIT, 3PM EST: Our payment gateway has confirmed that everything's back to normal now. Thanks for your patience, all! I'm sorry about the hassle.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark2014-10-14 05:38 pm

Upgrading against the POODLE vulnerability

Hi all,

Today another SSL vulnerability was announced. This one is named POODLE and is, while serious, much less serious than the Heartbleed event from some months ago.

Unfortunately, the only real way to fix the problem is to disable something called "SSLv3" entirely. Basically, this means that we instruct our servers that they are no longer allowed to speak version 3 of the SSL protocol (you can think of it as a language -- we ban this language from our servers). It turns out this is generally OK since most browsers don't actually speak using SSLv3 these days -- you actually use what's called TLS, which is a more modern, better way of protecting the stuff you send across the Internet.

The SSLv3 protocol is actually around 15 years old at this point, and TLS has been out so long that nearly every browser out there supports it. However, shutting off SSLv3 does mean that very old browsers -- IE6, for one -- can no longer talk to Dreamwidth using encryption. In this case, since the encryption wouldn't actually mean anything, we think it's better to not even pretend that it works.

I will be making this change sometime in the next hour or three. This really should impact almost none of you, but there might be one or two and, in that case, I'm sorry. We think it's better to do this so you know you're not actually secure than to let Dreamwidth pretend to be secure.

Edit: This has been deployed. SSLv3 is disabled on Dreamwidth.

Comments and questions welcome, as always!

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark2014-09-04 02:14 pm

DNS change today

Hi all,

We had a brief outage this morning. The cause was an (unexpected) policy change by our DNS provider, Dyn, deciding to shut us off. They had to roll back the change for unrelated reasons so we were back online, but it does mean that we need to migrate off of their service.

ETA: The policy change was that, for about 10 years now -- as long as I've been using Dyn! -- they had no usage/quota limits on their DNS service. Given that DNS requests are tiny and easy to serve, this made sense. They made a business decision recently to establish some (rather tiny) quotas. We're ... quite in excess of them (by some 15,000%) and we don't want to pay in excess of $500 USD/month for DNS service. Amazon's price is 10% of that. They probably tried to contact us, but I don't recall seeing any emails. Anyway, that's it; it's nothing particularly nefarious.

We will be moving our DNS service to Amazon's Route53 service. This kind of migration is fairly easy technically, but if there are problems it will probably mean Dreamwidth will be offline until they can be resolved. And, given the nature of how DNS works, it means that any outage will probably be measured in hours rather than minutes.

I've done my best to ensure that the changeover will go smoothly. If anything happens, though, we'll be on our [twitter.com profile] dreamwidth account to keep everybody apprised of the progress.

The switch will be flipped around 3:30pm PDT / 2230 UTC today, this is in about 90 minutes.

denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-08-12 12:20 pm

(no subject)

Some people are reporting having trouble reaching the site, especially from the UK/EU. Unfortunately, after looking into it, it seems to be an internet routing problem and not something we can fix on our end. Hang tight; our hosting service is pretty good at working around those sorts of problems pretty quickly.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-08-10 11:53 pm

(no subject)

We'll be starting the code push relatively soon; there may be a few blips here and there as we work. I'll update this post when it's all finished!

EDIT: Code has been pushed. Let us know if you encounter any problems! A list of the (many) bugfixes included in this push will be forthcoming.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-08-06 02:21 pm

Code push this weekend

We will be doing a code push this weekend: 9PM PDT Sunday 10 Aug/midnight EDT/5AM GMT Monday 11 Aug/noon PHT. (See in your time zone.)

This push will almost entirely consist of a (very large!) number of fixes for the mobile-friendly styles project, and should fix most of the remaining outstanding issues people have reported.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-07-14 12:23 am

(no subject)

Code push is complete and we're monitoring for any issues. If you spot something that looks off, let us know! The changes covered in this push are in this code tour.

(The site may be a bit sluggish for the next 20 minutes or so while the caches warm back up -- you don't have to tell us about that!)

Notification delays:

An update was posted to [site community profile] dw_news slightly before 0830 EST (see in your time zone). Comment notifications may be delayed for up to an hour or two, due to the high volume of notifications generated by each news post. Please don't worry about missing notifications until at least 1030 EST.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-07-13 11:50 pm

(no subject)

Our code push will begin in approximately 15 minutes. We'll let you know when it's complete!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-07-05 08:31 am

Code push next weekend

We will be doing a code push next weekend: 9PM PDT Sunday 13 Jul/midnight EDT/5AM GMT Monday 14 Jul/noon PHT. (See in your time zone.) This one is mostly bugfixes and small tweaks and shouldn't be much risk for things going kerblooey, and I really hope saying that didn't just jinx us.

(Getting one more code push in before I'm out who knows how long for surgical recovery!)
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-06-10 12:47 am

Code push underway

Tonight's code push is underway! We'll update this entry when we're done.

EDIT: And we're done! As always, we're watching for issues, but let us know of any problems.

Known issues:

* A [site community profile] dw_news post was posted just before 0800 EDT (see in your time zone). Comment notifications may be delayed for up to an hour or two, due to the high volume of notifications generated by each news post. Please don't worry about missing notifications until at least 1000 EDT.

* If you have a custom journal style, and you're seeing entries in the site default (Practicality/Neutral Good) rather than your custom layout, please recompile your layout layer and then try again. (This code push included S2 changes; we recompiled all the site styles, but we can't force custom styles to recompile.)
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-06-08 10:00 pm

Code push tomorrow night

We will be doing a code push tomorrow night, beginning between 9PM and 10PM PDT (midnight-1AM EDT, 5am-6am GMT, View in your time zone.) We'll announce when we're beginning the part that's visible to y'all.

The biggest change in this push is a new and improved frontend for community administration tasks, but there's a bunch more! We'll update you shortly after the push with what new stuff you can expect.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise2014-04-08 08:16 pm

"Heartbleed" security vulnerability

For those who have seen reference today in the press to the "Heartbleed" security vulnerability in OpenSSL, we'd like to reassure you that although we (like a large portion of the internet) were running the affected software, we patched our servers last night and were no longer vulnerable from that point.

We have no reason to believe that anyone was exploiting this vulnerability against us or that any user data has been compromised. We'll be changing our security certificates for extra confidence.

On the other hand, the nature of this vulnerablity means that it's impossible for a website to know for absolute certain whether someone was exploiting it. If someone was exploiting the vulnerability, against us or against any other website, they potentially have access to any information you sent to the site, including your username/password for the site and any data you sent to the site under HTTPS. It's a good idea to change your passwords pretty much everywhere, but don't do it until you can verify that a site is no longer vulnerable.

If you have any questions, feel free to ask!
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark2014-01-26 04:06 pm

Database maintenance

Hi all!

Today we started getting some alerts on the database, so I'm going to do some maintenance to verify the health and wellness of the machine and get it back to a non-alerting status.

I should be able to do this without any downtime, but just in case, you might want to make sure to use your favorite text editor to save a copy of any long entries or comments you're working on.

Once I've got things sorted out, I'll update this with more details for the technically curious.

Update [4:50PM PST]: sb-db06 (the slave) has been rebooted and is recovering, I'm doing system updates on it since the problem looks like a kernel bug (it struck both databases at the same time). Next: master failover then recover the other database.

Update [5:05PM PST]: I'm doing what we call a "master failover" now. This means I'm shifting all traffic from the database that was active (sb-db05) to the spare database (sb-db06). I have to shut off "extra" services like imports, feeds, and searches while this happens.

Update [5:30PM PST]: Well, that was unexpectedly bumpy. Sorry for that. There should be no further bumping, as we're now on the spare database so I can take maintenance on the original master.

Update [6:20PM PST]: If you had userpics not loading, they should be back to normal.

alierak: (Default)
[personal profile] alierak2014-01-26 11:45 am

Database restart

Our database servers have been generating some alerts today, where the monitoring system can't log into the servers to check on them. I can't log in either, but as best I can tell the databases are still running, or you wouldn't be seeing this. There is probably an issue with memory usage or excessive disk I/O, but it's kind of hard to troubleshoot at the moment. I would expect DW to be down for a while at some point today in order to restart the databases and/or reboot the servers. More info when [staff profile] mark's had a chance to look.