AWS outage

[alierak]

DW is seeing some issues due to today's Amazon outage. For right now it looks like the site is loading, but it may be slow. Some of our processes like notifications and journal search don't appear to be running and can't be started due to rate limiting or capacity issues. DW could go down later if Amazon isn't able to improve things soon, but our services should return to normal when Amazon has cleared up the outage.

Edit: all services are running as of 16:12 CDT, but there is definitely still a backlog of notifications to get through.

Edit 2: and at 18:20 CDT everything's been running normally for about the last hour.

Comments

[crashmargulies]

Thank you for keeping us updated! 😸

[chocolatefrogs]

Hmm that would explain why amazon prime music isn't working on my desktop, though it is on my phone.

[omnipotentwriter]

Thanks for the update! Will notifs go through once the process is running again, or are they lost forever?

[alierak]

Probably some of each. Notifications go through a series of processing steps with different queues and worker processes, like an assembly line. I can see jobs accumulating in some of the queues where no worker process is running, but I've also seen some errors from things that are running but are unable to contact bits of Amazon infrastructure. Anything that just sits in a queue will probably be fine, but I would assume some notifications have been dropped on the floor by malfunctioning robots, so to speak.

[alicedyer]

i love this explanation so much, thank you! and for the post as well!

[yarnandglue]

Thanks for keeping us posted!

[profiterole_reads]

Ah, that's why I noticed some answers I hadn't been notified about. Hopefully, it'll work again soon.

[gwendraith]

Thanks for the info. I wondered why notifications weren't coming through and my supermarket wouldn't let me check out :)

[yourlibrarian]

Sorry this is causing more work on DW's end! Does explain all the errors on Amazon this morning.

[huxleyenne]

Good luck addressing the issue. I hope all goes as smoothly as it can.

[ninetydegrees]

Thanks for the heads up!

[garryowen]

Thanks for the update!

[archaeologist_d]

There was lots of outages. I had a hard time with several sites.

[paserbyp]

DM using Amazon cloud service?

[commodityfetishx]

sounds like yes according to the answer to the comment below this one.

[thunderclan_territory]

Maybe I'm dumb, but what does Amazon have to do with Dreamwidth?

[archersangel]

Amazon Web Services, Inc. is a subsidiary of Amazon that provides on-demand cloud computing platforms and APIs to individuals, companies, and governments, on a metered, pay-as-you-go basis.

AWS is the thing that was having outages today & everyone who has been using them (which was a lot of people) had issues today.

[azurelunatic]

Adding to what archersangel said, Dreamwidth is running a lot of things on the AWS platform, since it's less labor intensive and expensive than owning the same amount of resources in an old-fashioned data center.

[andrewducker]

I'm intrigued that it's less expensive. I was under the impression that AWS was generally significantly more expensive than running something yourself, so long as you didn't have to worry about sudden scaling, and you had people who could maintain it fairly easily.

Is it the cost of the admins that would make it infeasible, or are there things where AWS is actually cheaper than running it locally?

[azurelunatic]

"People who could maintain it more easily" is probably the sticking point, since both mark and ailerak have day jobs and families. Although some of this is speculation and I am happy to be corrected.

[andrewducker]

Yeah, on the budget that Dreamwidth has, at its scale, it probably makes more sense to outsource that.

[thunderclan_territory]

Thanks for clearing this up, everybody! I get it now.

[havocthecat]

Thanks for the running updates!

[fennectik]

Thanks for the update.

Amazon DynamoDB Service Disruption in the Northern Virginia

[phd_ru]

This one, perhaps: https://aws.amazon.com/ru/message/101925/ ?