dw_maintenance | Notifications slow -- but recovering

Notifications slow -- but recovering

Hi all,

Due to some behind the scenes maintenance last night, our notifications system got delayed. I've fixed the issue now and it's working on catching up.

For details -- I've been experimenting with Kubernetes as a way to make managing production easier (and hopefully reduce costs!), but it turns out that one of our worker jobs that handles notifications doesn't use much CPU (it mostly spends time waiting on the database).

This caused the pod autoscaler to reduce the size of that particular deployment below what we needed to sustain throughput on our notifications service. The temporary fix is to pin that deployment size to something much larger, the better fix will be to integrate Kubernetes' pod autoscaler with the ability to monitor the queue depth on our task queue.

Sorry for the trouble, and thank you for the person who pinged us on Twitter. When I checked last night, everything was working, but as traffic came back up we fell behind and I wasn't watching anymore. My bad.

Flat | Top-Level Comments Only

Nowhere in the process of checking Amazon's SPF record when you receive mail from an [Bad username or site: amazonses @ com] sender will you find an include mechanism pointing to dreamwidth.org. The relevant identities (HELO and MAIL FROM) both belong to Amazon, and therefore so does the SPF record that you would be checking the sender IP address against. Amazon lists all the correct IP ranges in their SPF record, so it will pass (at the first receiving hop, obviously, they can't extend that guarantee to a forwarding situation like yours).

As far as I'm aware there is no reason to even look up Dreamwidth's SPF record in the process of receiving one of these emails. The From: header is not a relevant identity for SPF purposes; section 2.2 paragraph 2 recommends you don't check other identities besides the HELO and MAIL FROM, and section 11.2 (the only actual mention of the From: header) explains that it is easily falsified and unprotected by the authorization / authenticity assurance that SPF provides.

If you somehow ever do receive an email that is truly from dreamwidth.org and not amazonses.com, then you should look up our SPF record and interpret the include mechanism, but that's not what's currently happening.

> Nowhere ... will you find an include mechanism pointing to dreamwidth.org

Correct.
I did not claim that Amazon SES points to dreamwidth.org

I claimed that dreamwidth.org DNS points to amazonses.com:

dreamwidth.org text ="v=spf1 include:amazonses.com ~all"

> there is no reason to even look up Dreamwidth's SPF record in the process of receiving one of these emails

The reason why Gmail cares about dreamwidth.org SPF record when analyzing emails with From: ...[Bad username or site: dreamwidth @ org] -- is that Gmail needs to know if amazonses.com has permissions to send such ...[Bad username or site: dreamwidth @ org] emails in the first place.

Imagine that Gmail ignores dreamwidth.org SPF record, and then spammerparadise.com will start sending emails with From: ...[Bad username or site: dreamwidth @ org]
spammerparadise.com can legitimately claim that their IP address has permissions to send emails on behalf of spammerparadise.com
If Gmail did not check dreamwidth.org SPF record, then spammerparadise.com emails would have looked legitimate.

Unfortunately for spammerparadise.com, Gmail checks [Bad username or site: dreamwidth @ org] SPF and sees that only amazonses.com has permission to send emails with From: [Bad username or site: dreamwidth @ org]

Nope, you're contradicting the RFC again, and Google does not do what you say they're doing. At the risk of repeating myself, the From: header inside the message is not a relevant identity for SPF purposes.

> From: header inside the message is not a relevant identity for SPF purposes.

"From: header" (AKA "MAIL FROM") is the core concept in rfc7208:
~~~~~~~~
https://tools.ietf.org/html/rfc7208
In
particular, existing protocols place no restriction on what a sending
host can use as the "MAIL FROM" of a message or the domain given on
the SMTP HELO/EHLO commands.
This document describes version 1 of
the Sender Policy Framework (SPF) protocol, whereby ADministrative
Management Domains (ADMDs) can explicitly authorize the hosts that
are allowed to use their domain names, and a receiving host can check
such authorization.
~~~~~~~~

This is completely false. The MAIL FROM identity is not a header. It is an SMTP command issued during the SMTP conversation, after the HELO or EHLO command but before DATA. The From: header is part of the message that is sent after the DATA command. They have absolutely nothing to do with each other, and it is the MAIL FROM identity that is used by SPF, while the From: header is irrelevant.

Please stop replying with misinformation. Thanks.

Flat | Top-Level Comments Only

Notifications slow -- but recovering

Re: Dreamwidth.org SPF record

Re: Dreamwidth.org SPF record

Re: Dreamwidth.org SPF record

Re: Dreamwidth.org SPF record

Re: Dreamwidth.org SPF record