Notifications slow -- but recovering

[mark]

Hi all,

Due to some behind the scenes maintenance last night, our notifications system got delayed. I've fixed the issue now and it's working on catching up.

For details -- I've been experimenting with Kubernetes as a way to make managing production easier (and hopefully reduce costs!), but it turns out that one of our worker jobs that handles notifications doesn't use much CPU (it mostly spends time waiting on the database).

This caused the pod autoscaler to reduce the size of that particular deployment below what we needed to sustain throughput on our notifications service. The temporary fix is to pin that deployment size to something much larger, the better fix will be to integrate Kubernetes' pod autoscaler with the ability to monitor the queue depth on our task queue.

Sorry for the trouble, and thank you for the person who pinged us on Twitter. When I checked last night, everything was working, but as traffic came back up we fell behind and I wasn't watching anymore. My bad.

Comments

Re: Dreamwidth.org SPF record

[alierak]

Nope, you're contradicting the RFC again, and Google does not do what you say they're doing. At the risk of repeating myself, the From: header inside the message is not a relevant identity for SPF purposes.

Re: Dreamwidth.org SPF record

[dennisgorelik]

> From: header inside the message is not a relevant identity for SPF purposes.

"From: header" (AKA "MAIL FROM") is the core concept in rfc7208:
~~~~~~~~
https://tools.ietf.org/html/rfc7208
In
particular, existing protocols place no restriction on what a sending
host can use as the "MAIL FROM" of a message or the domain given on
the SMTP HELO/EHLO commands.
This document describes version 1 of
the Sender Policy Framework (SPF) protocol, whereby ADministrative
Management Domains (ADMDs) can explicitly authorize the hosts that
are allowed to use their domain names, and a receiving host can check
such authorization.
~~~~~~~~

Re: Dreamwidth.org SPF record

[alierak]

This is completely false. The MAIL FROM identity is not a header. It is an SMTP command issued during the SMTP conversation, after the HELO or EHLO command but before DATA. The From: header is part of the message that is sent after the DATA command. They have absolutely nothing to do with each other, and it is the MAIL FROM identity that is used by SPF, while the From: header is irrelevant.

Please stop replying with misinformation. Thanks.