karzilla: a green fist above the word SMASH! (Default)
Karzilla, Destroyer of Bugs ([staff profile] karzilla) wrote in [site community profile] dw_maintenance2016-08-07 02:24 pm

Resolved problem with emailed notifications

Sorry for the delayed response on the issue with emailed comment and support notifications not being delivered. We weren't receiving our emailed comment and support notifications saying that there was a problem...

We restarted the relevant software services around 10AM EDT, and the backlog of delayed notifications should have all been sent by now. Thanks for your patience, and extra thanks to everyone who tried to let us know about the problem!
wohali: photograph of Joan (Default)

[personal profile] wohali 2016-08-07 08:05 pm (UTC)(link)
Thanks for fixing things! Have you considered adding something like PagerDuty or Twilio (or even an IRC bot) to provide an alternative outgoing notification path?
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2016-08-07 08:12 pm (UTC)(link)

We have plenty of alerts, the length of the mail queue was just not one of them :)

kareila: IT prepares you for a life of fighting with PCs nonstop. (sysadmin)

[personal profile] kareila 2016-08-07 08:14 pm (UTC)(link)
I think [personal profile] alierak just starting monitoring the rate of outgoing emails and set an alert for when it drops to zero...