Notifications slow -- but recovering

[mark]

Hi all,

Due to some behind the scenes maintenance last night, our notifications system got delayed. I've fixed the issue now and it's working on catching up.

For details -- I've been experimenting with Kubernetes as a way to make managing production easier (and hopefully reduce costs!), but it turns out that one of our worker jobs that handles notifications doesn't use much CPU (it mostly spends time waiting on the database).

This caused the pod autoscaler to reduce the size of that particular deployment below what we needed to sustain throughput on our notifications service. The temporary fix is to pin that deployment size to something much larger, the better fix will be to integrate Kubernetes' pod autoscaler with the ability to monitor the queue depth on our task queue.

Sorry for the trouble, and thank you for the person who pinged us on Twitter. When I checked last night, everything was working, but as traffic came back up we fell behind and I wasn't watching anymore. My bad.

Comments

Re: Dreamwidth.org SPF record

[alierak]

This is completely false. The MAIL FROM identity is not a header. It is an SMTP command issued during the SMTP conversation, after the HELO or EHLO command but before DATA. The From: header is part of the message that is sent after the DATA command. They have absolutely nothing to do with each other, and it is the MAIL FROM identity that is used by SPF, while the From: header is irrelevant.

Please stop replying with misinformation. Thanks.