DNS change today

[mark]

Hi all,

We had a brief outage this morning. The cause was an (unexpected) policy change by our DNS provider, Dyn, deciding to shut us off. They had to roll back the change for unrelated reasons so we were back online, but it does mean that we need to migrate off of their service.

ETA: The policy change was that, for about 10 years now -- as long as I've been using Dyn! -- they had no usage/quota limits on their DNS service. Given that DNS requests are tiny and easy to serve, this made sense. They made a business decision recently to establish some (rather tiny) quotas. We're ... quite in excess of them (by some 15,000%) and we don't want to pay in excess of $500 USD/month for DNS service. Amazon's price is 10% of that. They probably tried to contact us, but I don't recall seeing any emails. Anyway, that's it; it's nothing particularly nefarious.

We will be moving our DNS service to Amazon's Route53 service. This kind of migration is fairly easy technically, but if there are problems it will probably mean Dreamwidth will be offline until they can be resolved. And, given the nature of how DNS works, it means that any outage will probably be measured in hours rather than minutes.

I've done my best to ensure that the changeover will go smoothly. If anything happens, though, we'll be on our dreamwidth account to keep everybody apprised of the progress.

The switch will be flipped around 3:30pm PDT / 2230 UTC today, this is in about 90 minutes.

Comments

[murdering]

I'm curious to know, what was the policy change? And why didn't they announce it?

(Was it the fan fic? It's always the fan fic.)

[denise]

They implemented a limit on queries per month, and we, uh, kind of overflowed it. A lot.

[murdering]

Whoops. Well, at least it wasn't the smut...

[azurelunatic]

Now I am wondering what percentage of DW's hits per month are smut-related.

[starwatcher]

.
I don't understand 'queries'. Is that making a support request, or just posting a comment? I'm thinking, posting a comment = "asking" the system to let it through? But I know zippo about the workings of the internet.
.

[mark]

DNS is the phone book of the Internet, basically.

When you type in 'dreamwidth.org' in your browser, your computer has to actually figure out -where- Dreamwidth is. It uses DNS to do that. So, a 'query' happens every time a computer asks the DNS servers 'hey, where is this "dreamwidth.org" I hear about?' and then the servers respond with an IP address (an Internet phone number).

[starwatcher]

.
Thank you; I appreciate the explanation.

(I see Denise explained to someone else, but I didn't grok that 'where is this' = 'query'. My 'duh' moment for the day.)
.

[denise]

When you put, say, "www.dreamwidth.org" into your web browser, your ISP sometimes has to look up where that address lives on the internet -- like if you had a business's name, and needed to know their address, so you looked them up in the phone book. One "hey, where does this site live" = one query. Now, your ISP remembers addresses for a little while, so it's not "one page load = one DNS query", but your ISP is also paranoid about people moving without telling it, so it looks up addresses pretty frequently!

[starwatcher]

.
A reply from Mark and Denise! I feel special. </snicker>

Thank you; that makes a lot of sense. One of my favorite things about DW (among so many) is how the tech folks manage to explain things in simple terms for those who are very non-tech.
.

[calmyourshit]

I'm curious what the policy change was. Is that something you can tell us?

[denise]

They implemented a limit on queries per month. We ... went past that limit. By a lot.

[calmyourshit]

I'm aware that "we broke the internet" is a pretty absurd reaction to that, because I have a (very low-level) understanding of how the internet works, but I'm going to say it anyway because it makes me happy: WE BROKE THE INTERNET. AWWWW YISSSSSS.

[momijizukamori]

Out of curiosity, what was the policy change, if you're free to say?

EDIT: AND THEN TWO OTHER PEOPLE BEAT ME TO THE QUESTION, hahaha

[denise]

Limit on queries per month! That we surpassed. Boy, howdy, did we surpass it.

[korralate]

I'd like to know what the change was too, if you're able to tell us.

[denise]

They instituted a limit on queries per month. We kind of went past it. A lot.

[korralate]

Ah! Thank you for telling us. Darn those pesky overages... Here's to hoping the transfer goes smoothly!

[princessofgeeks]

Sorry you guys have to deal with this! THanks for the headsup.

[ravengown]

Thank you for the heads up! Hopefully DW won't go down completely-- but it's too bad they changed their terms on you guys.

[sarajayechan]

That sucks about Dyn, but I'm glad you found a better server. Hopefully they won't put any tiny quotas in place!

[mark]

Amazon's business model is a lot better: basically they just set a realistic cost-per-query and charge you for all queries. There is no "free quota" level. You just pay. But the cost is very, very reasonable.

Dyn's model is that of Enterprise. They used to offer the bare-bones service we used which was pretty cheap and had no quotas. They've tried to get us to upgrade to Enterprise before, but it's super expensive and Rolls-Royce-esque. We don't need a Rolls. I'm more a BMW guy myself. Something solid, good, reasonably cheap, and gets the job done. Actually that probably better describes a Honda.

Anyway.

So, yeah, Amazon will be lots better. They've got a long track record of how they manage their pricing -- it never goes up, if anything it can be counted on to go down. Even if it doesn't, I'd much rather a pay-per-query model that is predictable.

[alittleshit]

Thanks guys for the quick explanations. I'm always amazed and how good you are when it comes to communicating with your userbase. /tips hat

[redsixwing]

Thanks for the heads up.

(ahaha we broke the internet)

[marilynesque]

Will replying to comments through email still work, or is that something we should refrain from?

[denise]

The mail servers use the same DNS as the web servers. Emailed-in comments may be delayed a bit, but they should get there eventually. (Really, the best answer is 'it depends'. It couldn't hurt to save a copy of the sent mail or CC yourself, just in case.)

[marilynesque]

Okay! Thanks Denise!

[ivorygates]

Is the new server service affiliated with Amazon.com?

[denise]

Route 53 is owned and operated by Amazon, yes. (To clarify, though: none of this involves user data in any way. DNS is the internet's phone book, essentially, keeping track of where you should go when you type a domain name into your browser's address bar -- when you type in, say, dw-maintenance.dreamwidth.org, your ISP looks at its own table and sees if it knows where that is, and if it doesn't, it calls up your DNS provider and says "yo, where do I find this address". We're switching the place that your ISP calls to find out where it finds that address.)

[ivorygates]

Thanks!

[garryowen]

Thanks for taking care of the problem so quickly. Awesome as always.

[jazzyjj]

Working as good as new on my end, at least thus far it is. That's one of the things I like about Dreamwidth; you folks really do care about keeping your customers in the loop and it shows. I'm not going to bother asking why this decision was made, since that's been explained here more than once. In addition, it's a bit too technical for my blood. Guess I'd better go and finish my dinner. I've not been feeling the greatest the past few days.

[dil]

And why don't you run DNS on the same machines as web servers? What's the problem with it?

[mark]

I used to run my own DNS and for a while I think we even self-hosted Dreamwidth's DNS. The main advantage to not self-hosting though is that, if we lose our web server provider, we can do things like point the web site at a new IP so people can get a status page up. Also, in the unfortunate case of someone attacking the site (DDoS, etc) it's really nice to have the DNS service run on someone's very powerful hardware and very fat pipes -- ours certainly aren't that big.

A second useful thing about it is that folks like Amazon actually have a widely distributed DNS infrastructure. Every DNS request is going to take some time to complete, which adds to how long it takes the site to load for people. If we hosted DNS ourself then our users who aren't near Texas, USA will have extra latency. Amazon has servers all over the globe and this helps the site feel faster for people who aren't near us.

Also personally, it's well worth the $50 a month to have someone else worry about DNS infrastructure. They will actually do it much better than I could in the time I have available. (And I'd rather spend that time on DW itself!)

DNS

[babysprite]

Understood

If you don't have a robust set of resources, better to let someone who does provide the DNS.

Time to renew my paid subscription to help DW pay the bills.【ツ】

DNS costs DW fifty dollars a month, I can subscribe for fifty dollars a year.

Done.

[arethinn]

$50 a month for ~119 million DNS lookups? Wow, they're charging tiny fractions of a cent per each.

[mark]

Yup. 40 cents USD per million queries; it's pretty cheap. Realistically though, DNS is super cheap to serve. The expense/hard part is in hosting servers in dozens of regions to minimize latency, not the hardware cost itself.

[al_zorra]

NOT HAPPY giving amazilla any business at all. They treat labor shamefully and are determined to own everything.

Just sayin.

[thrashmetals]

Then I really hope you don't shop at Walmart or most retail outlets.

[ranunculus]

Gotta agree that I'm really not happy going to Amazon, and no I don't shop at Amazon, Walmart or Mc Donald's.

[al_zorra]

In fact, I do not.

[denise]

I'm not tremendously happy with a lot of Amazon's business practices either, but there are very few options at our volume that are not either ruinously expensive, unreliable enough to lead to regular downtime, or slow enough to cause a noticeable lag in the site's response time.

business practices

[babysprite]

Understood and appreciated.

Some people have to shop WalMart because they can't afford anywhere else.

Others have to buy from Amazon.com because they're home-bound and can't leave their residences.

/\__/\
(='.'=)
(")_(")

babysprite

[linaelyn]

You guys rock. Thanks for finding solutions when problems arise.

[shanghaied]

Thank you for looking after us! It's this sort of caring that made Dreamwidth the right place to migrate to :-)

[weofodthignen]

So that's what that was. Thanks for telling us about it :-)

[azurelunatic]

Looks like it went smoothly!

Back when I was working tier 1 tech support for a local domain registrar, I told my supervisor that I was reading up on DNS in order to do my job better. Of course I'd poked at zone files in college just like most of my friends, but I wanted to read up now that I'd be dealing with end-users who had probably done terrible things to their A Record. My supervisor shuddered.

"You know how you have two evil magicians battling in a room somewhere?" he asked me. I nodded. "And that stuff that falls to the floor when they're fighting?" I nodded. "That's DNS."

It's been a couple years, so I guess if I finish the new Seanan book quickly, I know what the rest of my leisure reading's going to be!

[mark]

Hahahaha, that's pretty accurate. It really is amazing the Internet keeps working.

[cdybedahl]

DNS is an amazingly resilient and forgiving system. You can set it up horribly, horribly wrong and it'll still work well enough that most users (and most sysadmins) won't notice any problem.

[azurelunatic]

If you feel like telling some war stories about terrible DNS things you have seen in the line of work, I know at least one person who would be raptly entertained.

[cdybedahl]

Since we scan the domains of others from the outside, we don't really get any good stories. Quite a few "WTF, how did they manage _that_?", but they're usually pretty obscure and not at all funny.

[gwendraith]

Thank you for the heads up, it looks like it all went well :)

[sharpiefan]

I'm gonna say, i'm reading this on the 5th September at 11:45 BST (10:4 GMT/UTC) so it looks like it all flipped over just fine.

Cheers for letting us know, even though you got very little notice and we had less. Moving servers, hosts and what-not is no fun.

*HUGS ALL THE DW STAFF*

[decepticon_mistress]

I'm reading this on September 5th and man, that sucked about the policy change and you guys had to switch over to a new service. =/ Hopefully Amazon's service will work out the best for DW from here on out!

[dickgrayson]

Thank you for the info and good luck with the change! :D

Edit: Well today is the 5th so it seems all went well and easy!

[cdybedahl]

If you have any questions about DNS or want advice on how it should be set up, please feel free to ask. In my day job I write quality-checking software for DNS zones (including some of that which is being used to test new top-level domains before they are lot loose on the Internet).

If you just want to check that everything is all right, we (as in .SE, the Swedish ccTLD registry) have a web interface intended for public use at http://dnscheck.iis.se/

[ofdrumsthesound]

My inbox is suddenly freaking out. Not sure if it's related, though.

[misskat]

Can you define "freaking out" for me? Did it suddenly turn upside down, or you can't delete messages, or it moved to Minnesota to be an insurance agent?

[ofdrumsthesound]

When I try to delete messages they're replaced with other deleted messages. Not sure if it moved to Minnesota or not LOL

[denise]

That sounds like your browser is holding on to its cache! Try clearing your browser cache and that should clear it up.

[ofdrumsthesound]

I just did that a day or two ago . . . I'll try it again.

[denise]

If you're regularly having problems with your browser showing you outdated versions of pages, your browser might need to have its settings adjusted to check for new versions of the page with every page load. How you do that depends on the browser, but it should be somewhere in the preferences!

[ofdrumsthesound]

Thanks :D