karzilla: a green fist above the word SMASH! (Default)
Karzilla, Destroyer of Bugs ([staff profile] karzilla) wrote in [site community profile] dw_maintenance2016-08-07 02:24 pm

Resolved problem with emailed notifications

Sorry for the delayed response on the issue with emailed comment and support notifications not being delivered. We weren't receiving our emailed comment and support notifications saying that there was a problem...

We restarted the relevant software services around 10AM EDT, and the backlog of delayed notifications should have all been sent by now. Thanks for your patience, and extra thanks to everyone who tried to let us know about the problem!
brave_heart_verity: (peonies and stripes)

[personal profile] brave_heart_verity 2016-08-07 07:37 pm (UTC)(link)
alexseanchai: Blue and purple lightning (Default)

[personal profile] alexseanchai 2016-08-07 07:40 pm (UTC)(link)
Yay working again!
1179875: (Default)

[personal profile] 1179875 2016-08-07 07:57 pm (UTC)(link)
Thank you! :D

[personal profile] dagbok 2016-08-07 07:57 pm (UTC)(link)
Thank you!
mildred_of_midgard: (Default)

[personal profile] mildred_of_midgard 2016-08-07 08:00 pm (UTC)(link)
Haha, that reminds me of this: https://twitter.com/devops_borat/status/291243959370518528

DevOps Borat

90% of devops are fail simple interview question of what is most critical piece of infrastructure. Is outgoing mail on Nagios server!


Glad it's fixed!
wohali: photograph of Joan (Default)

[personal profile] wohali 2016-08-07 08:05 pm (UTC)(link)
Thanks for fixing things! Have you considered adding something like PagerDuty or Twilio (or even an IRC bot) to provide an alternative outgoing notification path?
cxcvi: Red cubes, sitting on a reflective surface, with a white background (Default)

[personal profile] cxcvi 2016-08-07 08:10 pm (UTC)(link)
"We didn't get the email saying that email is down because email was down."

Glad to have it back, though.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2016-08-07 08:12 pm (UTC)(link)

We have plenty of alerts, the length of the mail queue was just not one of them :)

lurkerwolf: (Default)

[personal profile] lurkerwolf 2016-08-07 08:14 pm (UTC)(link)
Glad it's fixed now. It's understandable it took time to fix it when not even you guys got notifications.
kareila: IT prepares you for a life of fighting with PCs nonstop. (sysadmin)

[personal profile] kareila 2016-08-07 08:14 pm (UTC)(link)
I think [personal profile] alierak just starting monitoring the rate of outgoing emails and set an alert for when it drops to zero...
kore: (Default)

[personal profile] kore 2016-08-07 08:40 pm (UTC)(link)
We weren't receiving our emailed comment and support notifications saying that there was a problem...

Oh, DEAR.
rydra_wong: dreamsheep with spork and "SheepSpork" logo; no, it wouldn't make any more sense if you saw it  (dreamwidth -- sheepspork)

[personal profile] rydra_wong 2016-08-07 08:41 pm (UTC)(link)
We weren't receiving our emailed comment and support notifications saying that there was a problem...

BUT WE WERE WAVING. WHY DID YOU NOT SEE US WAVING.

(Thanks for getting it fixed!)
azurelunatic: A glittery black pin badge with a blue holographic star in the middle. (Default)

[personal profile] azurelunatic 2016-08-07 09:59 pm (UTC)(link)

It slowly dawned on me that I hadn't seen any official notice taken, but the clue took a while to arrive in the post-surgical haze.

Perfect storms and all that.

hokuton_punch: (hikaru no go sai squee)

[personal profile] hokuton_punch 2016-08-08 02:12 am (UTC)(link)
Thank you so much! My inbox was feeling like a sad, empty ghost town. I'm glad it was a quick fix.

[personal profile] jazzyjj 2016-08-08 03:07 am (UTC)(link)
No wonder my work email inbox just saw an influx of these requests, *grin*. Glad things are fixed though for those who were affected. I'm off to bed now. Shutting this thing down in...
sarasa_cat: (Default)

[personal profile] sarasa_cat 2016-08-08 05:38 am (UTC)(link)
Thanks for fixing this!
gwendraith: please don't take (yay cat)

[personal profile] gwendraith 2016-08-08 08:25 am (UTC)(link)
Thank you for fixing it promptly as soon as you knew there was a problem :)
filialucis: (Default)

[personal profile] filialucis 2016-08-08 08:34 am (UTC)(link)
Thanks for sorting it out!
mirrorofsmoke: The words "We are Groot" and a picture of Baby Groot on an icon with a swirly galaxy background. (Default)

[personal profile] mirrorofsmoke 2016-08-11 06:31 pm (UTC)(link)
We weren't receiving our emailed comment and support notifications saying that there was a problem
Oh man, that's really kind of meta isn't it? Maybe that's the wrong word, but thank you for the fix. I was about to go leave a support request for it when I found that hey a bunch of people beat me to it. :D
poppy100: (Default)

[personal profile] poppy100 2016-08-16 02:24 am (UTC)(link)
Wow..