Notifications and some issues due to AWS outage

[mark]

Hi all --

Due to the major AWS outage today, some of our systems are impacted -- particularly around notifications and emails. We will work to make sure that service is restored as soon as we can.

Unfortunately, due to the nature of the outage, it may be that notifications were lost along the way. We don't really have a way to go back and re-identify notifications that should have been sent and weren't, so the best bet is to check back on any conversations you were in or your recent posts to see if you missed anything.

Once service is fully restored (which should be soon), though, notifications should be back to their usual selves.

Edit: We are starting to see things moving again! Notifications are being generated and it looks like we may not have lost anything and they're just going to be quite tardy. Knock on wood.

Comments

[muccamukk]

Thanks for the update! I was wondering what had happened.

[kingstoken]

Oh, this explains a lot.

Alternate Communication Management Strategy

[frith]

I suspect that if a comment or reply is associated with a community or blog, it can be found here: http://www.dreamwidth.org/comments/recent ? Otherwise, if it's a direct message, check your inbox for fresh messages here: https://www.dreamwidth.org/inbox/ ?

Re: Alternate Communication Management Strategy

[rampitec]

Does not seem to work. Inbox is also not populated with new comments, it is not just emails. Confirmed 5 min ago.

[yourlibrarian]

Cheers to the DW team impacted by this mess!

[luckyzukky]

thanks for updating ^^

[kore]

Thank you for keeping us in the loop, as always!

[dewline]

I must've been off-line when the outage happened. Good to have the news, though!

Re: Alternate Communication Management Strategy

[frith]

Curious. The Manage Comments (the first link) has updated for me, showing that I've commented here and that my comment has been replied to.

Re: Alternate Communication Management Strategy

[rampitec]

Interesting. First link works for me too, but not the second.

[assholic]

I just did a test PM to myself and I'm still a) not getting either the DW inbox notif for it or a gmail one, and b) it's not showing in my sent tab of my DW inbox.

How ongoing of a timeframe should we expect, ballpark?

[thistleingrey]

Thanks for updating! Good to know.

Re: Alternate Communication Management Strategy

[frith]

I can't tell if the Inbox link is updating since I haven't made a blog post in 5 days and the last time I got a DM was 87 weeks ago. Suffice to say I don't check my Inbox that often. ^_^

On a side note, thanks to Snopes, I believe the AWS outage here is for Amazon Web Services. https://snopes-feed.dreamwidth.org/8075835.html

Re: Alternate Communication Management Strategy

[assholic]

Inbox isn't working at all. Not even for PMs. I've not only tried to send out to others, I've tried to send to myself and they're all not being sent or received. Send yourself a PM and see what happens.

EDIT: Now I'm getting a flood of my test PMs. YAY!

[paserbyp]

An outage at Amazon Web Services' us-east-1 cloud region is impacting customers globally.

The company's Northern Virginia region had issues with EC2 instances in September after resource contention problems. The same region also had issues in November 2020 after the Kinesis service experienced API errors and caused a number of AWS offerings to struggle.

[pantswarrior]

I expect from the info given, there's nothing DW staff can do about it. I was dealing with this myself at work today (my org also uses AWS for some things), we used this status page to get updates. Looks like the last update was about half an hour ago and they say the primary issues are fixed, but they're working on all the further issues caused by those issues.

[paserbyp]

Amazon subsidiaries like IMDb and Ring went down, as did games like Player Unknown's Battlegrounds, Valorant, Clash of Clans, Destiny 2 and Dead by Daylight, amongst others. The issue began at around 10:45AM ET.

Other companies impacted include stock and crypto platform Public.com, NFT site RECUR, adult webcam site CB Explorer, tech news website Protocol, audio and video program ReelCrafter v2, software platform ConnectWise, Splunk's Rigor, M1 Finance, Duolingo, and more.

Users of Disney+, League of Legends, Tinder, Instacart, Venmo, Robinhood, Roku, Kindle, and the McDonalds app are also reporting problems.

[assholic]

Good to know. Thank you for the link. That's super helpful and I'll probably be keeping an eye on it for later. I didn't realize something like that would have the effect the news is printing about how widespread this is and how slow to update the original service was. I'm surprised with multiple issues like this in a year, they haven't figured out some kind of emergency backup system. Seems pretty fingers crossed to me.

[mistressofmuses]

Thank you for the update!

[darkestabsol]

Oh, thanks for the updates!

Re: Alternate Communication Management Strategy

[rampitec]

Seem to work as of now at least for me.

[badly_knitted]

I was wondering about this last night, but it was late so I figured I'd leave it and see how things were this morning because you guys are always on the ball with problems. Glad to hear it's getting sorted, thank you!

[thewayne]

Interesting. I received similar emails at the university where I work that the outage clobbered a online testing subsystem, which isn't good with finals fast upon us!

Thank you so much

[akatsija]

I appreciate your work and the prompt updates you provide.

And no, no sarcasm was implied. It was an expression of sincere gratitude.

--
А.Кац и Я

[casimirian]

Thank you, Mark. I received a notification today about warp bubbles and was so pleased and excited to read the article. I give you and Dreamwidth 100% of the credit for the notification and the scientific development. Congratulations!

[marianme]

Thank you for keeping us informed and from all the work you do. I really enjoy my little community in Dreamwidth.