DNS change today

[mark]

Hi all,

We had a brief outage this morning. The cause was an (unexpected) policy change by our DNS provider, Dyn, deciding to shut us off. They had to roll back the change for unrelated reasons so we were back online, but it does mean that we need to migrate off of their service.

ETA: The policy change was that, for about 10 years now -- as long as I've been using Dyn! -- they had no usage/quota limits on their DNS service. Given that DNS requests are tiny and easy to serve, this made sense. They made a business decision recently to establish some (rather tiny) quotas. We're ... quite in excess of them (by some 15,000%) and we don't want to pay in excess of $500 USD/month for DNS service. Amazon's price is 10% of that. They probably tried to contact us, but I don't recall seeing any emails. Anyway, that's it; it's nothing particularly nefarious.

We will be moving our DNS service to Amazon's Route53 service. This kind of migration is fairly easy technically, but if there are problems it will probably mean Dreamwidth will be offline until they can be resolved. And, given the nature of how DNS works, it means that any outage will probably be measured in hours rather than minutes.

I've done my best to ensure that the changeover will go smoothly. If anything happens, though, we'll be on our dreamwidth account to keep everybody apprised of the progress.

The switch will be flipped around 3:30pm PDT / 2230 UTC today, this is in about 90 minutes.

Comments

[murdering]

I'm curious to know, what was the policy change? And why didn't they announce it?

(Was it the fan fic? It's always the fan fic.)

[denise]

They implemented a limit on queries per month, and we, uh, kind of overflowed it. A lot.

[calmyourshit]

I'm curious what the policy change was. Is that something you can tell us?

[denise]

They implemented a limit on queries per month. We ... went past that limit. By a lot.

[momijizukamori]

Out of curiosity, what was the policy change, if you're free to say?

EDIT: AND THEN TWO OTHER PEOPLE BEAT ME TO THE QUESTION, hahaha

[denise]

Limit on queries per month! That we surpassed. Boy, howdy, did we surpass it.

[korralate]

I'd like to know what the change was too, if you're able to tell us.

[denise]

They instituted a limit on queries per month. We kind of went past it. A lot.

[princessofgeeks]

Sorry you guys have to deal with this! THanks for the headsup.

[ravengown]

Thank you for the heads up! Hopefully DW won't go down completely-- but it's too bad they changed their terms on you guys.

[sarajayechan]

That sucks about Dyn, but I'm glad you found a better server. Hopefully they won't put any tiny quotas in place!

[mark]

Amazon's business model is a lot better: basically they just set a realistic cost-per-query and charge you for all queries. There is no "free quota" level. You just pay. But the cost is very, very reasonable.

Dyn's model is that of Enterprise. They used to offer the bare-bones service we used which was pretty cheap and had no quotas. They've tried to get us to upgrade to Enterprise before, but it's super expensive and Rolls-Royce-esque. We don't need a Rolls. I'm more a BMW guy myself. Something solid, good, reasonably cheap, and gets the job done. Actually that probably better describes a Honda.

Anyway.

So, yeah, Amazon will be lots better. They've got a long track record of how they manage their pricing -- it never goes up, if anything it can be counted on to go down. Even if it doesn't, I'd much rather a pay-per-query model that is predictable.

[alittleshit]

Thanks guys for the quick explanations. I'm always amazed and how good you are when it comes to communicating with your userbase. /tips hat

[redsixwing]

Thanks for the heads up.

(ahaha we broke the internet)

[marilynesque]

Will replying to comments through email still work, or is that something we should refrain from?

[denise]

The mail servers use the same DNS as the web servers. Emailed-in comments may be delayed a bit, but they should get there eventually. (Really, the best answer is 'it depends'. It couldn't hurt to save a copy of the sent mail or CC yourself, just in case.)

[ivorygates]

Is the new server service affiliated with Amazon.com?

[denise]

Route 53 is owned and operated by Amazon, yes. (To clarify, though: none of this involves user data in any way. DNS is the internet's phone book, essentially, keeping track of where you should go when you type a domain name into your browser's address bar -- when you type in, say, dw-maintenance.dreamwidth.org, your ISP looks at its own table and sees if it knows where that is, and if it doesn't, it calls up your DNS provider and says "yo, where do I find this address". We're switching the place that your ISP calls to find out where it finds that address.)

[garryowen]

Thanks for taking care of the problem so quickly. Awesome as always.

[jazzyjj]

Working as good as new on my end, at least thus far it is. That's one of the things I like about Dreamwidth; you folks really do care about keeping your customers in the loop and it shows. I'm not going to bother asking why this decision was made, since that's been explained here more than once. In addition, it's a bit too technical for my blood. Guess I'd better go and finish my dinner. I've not been feeling the greatest the past few days.

[dil]

And why don't you run DNS on the same machines as web servers? What's the problem with it?

[mark]

I used to run my own DNS and for a while I think we even self-hosted Dreamwidth's DNS. The main advantage to not self-hosting though is that, if we lose our web server provider, we can do things like point the web site at a new IP so people can get a status page up. Also, in the unfortunate case of someone attacking the site (DDoS, etc) it's really nice to have the DNS service run on someone's very powerful hardware and very fat pipes -- ours certainly aren't that big.

A second useful thing about it is that folks like Amazon actually have a widely distributed DNS infrastructure. Every DNS request is going to take some time to complete, which adds to how long it takes the site to load for people. If we hosted DNS ourself then our users who aren't near Texas, USA will have extra latency. Amazon has servers all over the globe and this helps the site feel faster for people who aren't near us.

Also personally, it's well worth the $50 a month to have someone else worry about DNS infrastructure. They will actually do it much better than I could in the time I have available. (And I'd rather spend that time on DW itself!)

[al_zorra]

NOT HAPPY giving amazilla any business at all. They treat labor shamefully and are determined to own everything.

Just sayin.

[thrashmetals]

Then I really hope you don't shop at Walmart or most retail outlets.

[linaelyn]

You guys rock. Thanks for finding solutions when problems arise.

[shanghaied]

Thank you for looking after us! It's this sort of caring that made Dreamwidth the right place to migrate to :-)

[weofodthignen]

So that's what that was. Thanks for telling us about it :-)

[azurelunatic]

Looks like it went smoothly!

Back when I was working tier 1 tech support for a local domain registrar, I told my supervisor that I was reading up on DNS in order to do my job better. Of course I'd poked at zone files in college just like most of my friends, but I wanted to read up now that I'd be dealing with end-users who had probably done terrible things to their A Record. My supervisor shuddered.

"You know how you have two evil magicians battling in a room somewhere?" he asked me. I nodded. "And that stuff that falls to the floor when they're fighting?" I nodded. "That's DNS."

It's been a couple years, so I guess if I finish the new Seanan book quickly, I know what the rest of my leisure reading's going to be!

[mark]

Hahahaha, that's pretty accurate. It really is amazing the Internet keeps working.

[gwendraith]

Thank you for the heads up, it looks like it all went well :)

[sharpiefan]

I'm gonna say, i'm reading this on the 5th September at 11:45 BST (10:4 GMT/UTC) so it looks like it all flipped over just fine.

Cheers for letting us know, even though you got very little notice and we had less. Moving servers, hosts and what-not is no fun.

*HUGS ALL THE DW STAFF*

[decepticon_mistress]

I'm reading this on September 5th and man, that sucked about the policy change and you guys had to switch over to a new service. =/ Hopefully Amazon's service will work out the best for DW from here on out!

[dickgrayson]

Thank you for the info and good luck with the change! :D

Edit: Well today is the 5th so it seems all went well and easy!

[cdybedahl]

If you have any questions about DNS or want advice on how it should be set up, please feel free to ask. In my day job I write quality-checking software for DNS zones (including some of that which is being used to test new top-level domains before they are lot loose on the Internet).

If you just want to check that everything is all right, we (as in .SE, the Swedish ccTLD registry) have a web interface intended for public use at http://dnscheck.iis.se/

[ofdrumsthesound]

My inbox is suddenly freaking out. Not sure if it's related, though.

[misskat]

Can you define "freaking out" for me? Did it suddenly turn upside down, or you can't delete messages, or it moved to Minnesota to be an insurance agent?