dw_maintenance | Notifications slow -- but recovering

Notifications slow -- but recovering

Hi all,

Due to some behind the scenes maintenance last night, our notifications system got delayed. I've fixed the issue now and it's working on catching up.

For details -- I've been experimenting with Kubernetes as a way to make managing production easier (and hopefully reduce costs!), but it turns out that one of our worker jobs that handles notifications doesn't use much CPU (it mostly spends time waiting on the database).

This caused the pod autoscaler to reduce the size of that particular deployment below what we needed to sustain throughput on our notifications service. The temporary fix is to pin that deployment size to something much larger, the better fix will be to integrate Kubernetes' pod autoscaler with the ability to monitor the queue depth on our task queue.

Sorry for the trouble, and thank you for the person who pinged us on Twitter. When I checked last night, everything was working, but as traffic came back up we fell behind and I wasn't watching anymore. My bad.

Flat | Top-Level Comments Only

I don't mean to be rude, but what part of that Received-SPF header do you think has anything to do with Dreamwidth?

Most likely Dreamwidth.org DNS is misconfigured (specifically, SPF record), and that is causing Gmail to classify emails that Amazon SES sends on behalf of Dreamwidth.org -- as "SPF: fail".

In order to make it right, Dreamwidth.org DNS should explicitly declare that Amazon SES IP addresses have permission to send emails on behalf of dreamwidth.org

I'm sorry, but that's not correct, and no part of the header you showed has anything to do with Dreamwidth. An SPF check is performed against the SPF record of the envelope sender's domain. In this case the Received-SPF header states that the sender is something at amazonses.com, so the relevant SPF record is Amazon's, not Dreamwidth's. If Amazon hasn't allowed the IP address from which you received the message, then it should be a fail.

In this case the IP address from which you received the message appears to be one of Google's, not one of Amazon's or Dreamwidth's. Neither Dreamwidth nor Amazon uses Google to send email, so the failure is correct and there is nothing we need to adjust.

1) Thank you - you are correct.

I see now that there are 2 "Received-SPF" headers in the email sent by [Bad username or site: dreamwidth @ org] that I received through [Bad username or site: gmail @ com]:
~~~~~~
Received-SPF: fail (google.com: domain of 0100016ee886fbff-ceef5459-337e-4a6a-a337-e936f2318a2d-000000@amazonses.com does not designate 209.85.220.69 as permitted sender) client-ip=209.85.220.69;
~~~~~~
Received-SPF: pass (google.com: domain of 0100016ee886fbff-ceef5459-337e-4a6a-a337-e936f2318a2d-000000@amazonses.com designates 54.240.11.151 as permitted sender) client-ip=54.240.11.151;
~~~~~~

"pass" record indicates that google.com recognizes 54.240.11.151 IP address (that belongs to Amazon SES) as a permitted sender).

"fail" record, probably (I am guessing here) refers to email forwarding feature inside of Gmail (forwarding from one email address to another).

I incorrectly assumed that this "fail" record indicates misconfiguration in dreamwidth.org

"pass" record really matters and indicates that dreamwidth.org DNS configured SPF record correctly.

I am sorry for misreading my email headers.

> the relevant SPF record is Amazon's, not Dreamwidth's

Are you sure it is a correct statement?

Wouldn't it be more correct to state that both Amazon DNS SPF and Dreamwidth DNS SPF together define whether specific sender IP address is permitted for sending emails on behalf of dreamwidth.org?

Specifically, Dreamwidth DNS SPF delegates defining email sending permissions to Amazon SES:
~~~~~~~~~~~~~~~~~
dreamwidth.org text =
"v=spf1 include:amazonses.com ~all"
~~~~~~~~~~~~~~~~~

Then Amazon SES uses these permissions to permit 54.240.11.151 IP address to send emails that originate from dreamwidth.org (and other domains that Amazon SES serves)
~~~~~~~~~~~~~
amazonses.com text =

"v=spf1 ip4:199.255.192.0/22 ip4:199.127.232.0/22 ip4:54.240.0.0/18 ip4:69.169.224.0/20 ip4:76.223.180.0/23 ip4:76.223.188.0/24 ip4:76.223.189.0/24 ip4:76.223.190.0/24 -all"
~~~~~~~~~~~~~

1) Thanks, apology accepted.

2) Yes, I'm sure it's a correct statement. Amazon isn't sending on behalf of dreamwidth.org; an email can only have one envelope sender address and in this case it's something at amazonses.com, so that is the only domain whose SPF record is relevant. That email could be from some other Amazon customer and the behavior would be the same. Amazon guarantees that emails sent thru SES will pass an SPF check by default, by using their domain to do the sending. It is possible to use a custom domain for the envelope sender in SES, and then you're responsible for the SPF record, but as far as I know we're not doing that.

There is another, completely different, use case for SPF records involved in DMARC, where the check would be performed against the domain of the address in the From: header instead of the envelope sender. That's the part where Dreamwidth's SPF record might be relevant, except that isn't what you asked about, and we aren't making any attempt to use DMARC as far as I know. That would also require using a custom envelope sender domain, since DMARC requires the domains of From: header and envelope sender to match.

I would recommend that you educate yourself further on these topics using the abundant documentation available on the web, as well as running your own email services to gain experience if it's something that really interests you. There also exist "header analyzer" services that will help you understand the various email delivery issues if you paste in the headers from an email you've received. All my efforts here aside, Dreamwidth is not one of those services and this

dw_maintenance post is about the processing of notifications, not email deliverability.

> an email can only have one envelope sender address and in this case it's something at amazonses.com

I agree here.

> so that is the only domain whose SPF record is relevant.

I disagree with that part.
If "From" email is "...[Bad username or site: dreamwidth @ org]" then dreamwidth.org SPF settings are relevant too, even if sender IP address belongs to Amazon SES (and does not belong to dreamwidth.org).

> Amazon guarantees that emails sent thru SES will pass an SPF check by default

... only if Dreamwidth DNS specifies something like

dreamwidth.org text =
"v=spf1 include:amazonses.com ~all"

Or do you mean that even without dreamwidth.org granting permissions to Amazon SES, Amazon SES will guarantee SPF check to pass?

> that isn't what you asked about

Correct. I am not referring to DMARC.
I am only referring to "v=spf1" DNS text record.

All right, best of luck with your disagreement with RFC 7208.

> your disagreement with RFC 7208

I do NOT disagree with RFC 7208.

RFC 7208 supports my position:
~~~~~~~~~
https://tools.ietf.org/html/rfc7208
The "include" mechanism triggers a recursive evaluation of
check_host().
~~~~~~~~~

Nowhere in the process of checking Amazon's SPF record when you receive mail from an [Bad username or site: amazonses @ com] sender will you find an include mechanism pointing to dreamwidth.org. The relevant identities (HELO and MAIL FROM) both belong to Amazon, and therefore so does the SPF record that you would be checking the sender IP address against. Amazon lists all the correct IP ranges in their SPF record, so it will pass (at the first receiving hop, obviously, they can't extend that guarantee to a forwarding situation like yours).

As far as I'm aware there is no reason to even look up Dreamwidth's SPF record in the process of receiving one of these emails. The From: header is not a relevant identity for SPF purposes; section 2.2 paragraph 2 recommends you don't check other identities besides the HELO and MAIL FROM, and section 11.2 (the only actual mention of the From: header) explains that it is easily falsified and unprotected by the authorization / authenticity assurance that SPF provides.

If you somehow ever do receive an email that is truly from dreamwidth.org and not amazonses.com, then you should look up our SPF record and interpret the include mechanism, but that's not what's currently happening.

> Nowhere ... will you find an include mechanism pointing to dreamwidth.org

Correct.
I did not claim that Amazon SES points to dreamwidth.org

I claimed that dreamwidth.org DNS points to amazonses.com:

dreamwidth.org text ="v=spf1 include:amazonses.com ~all"

> there is no reason to even look up Dreamwidth's SPF record in the process of receiving one of these emails

The reason why Gmail cares about dreamwidth.org SPF record when analyzing emails with From: ...[Bad username or site: dreamwidth @ org] -- is that Gmail needs to know if amazonses.com has permissions to send such ...[Bad username or site: dreamwidth @ org] emails in the first place.

Imagine that Gmail ignores dreamwidth.org SPF record, and then spammerparadise.com will start sending emails with From: ...[Bad username or site: dreamwidth @ org]
spammerparadise.com can legitimately claim that their IP address has permissions to send emails on behalf of spammerparadise.com
If Gmail did not check dreamwidth.org SPF record, then spammerparadise.com emails would have looked legitimate.

Unfortunately for spammerparadise.com, Gmail checks [Bad username or site: dreamwidth @ org] SPF and sees that only amazonses.com has permission to send emails with From: [Bad username or site: dreamwidth @ org]

Nope, you're contradicting the RFC again, and Google does not do what you say they're doing. At the risk of repeating myself, the From: header inside the message is not a relevant identity for SPF purposes.

> From: header inside the message is not a relevant identity for SPF purposes.

"From: header" (AKA "MAIL FROM") is the core concept in rfc7208:
~~~~~~~~
https://tools.ietf.org/html/rfc7208
In
particular, existing protocols place no restriction on what a sending
host can use as the "MAIL FROM" of a message or the domain given on
the SMTP HELO/EHLO commands.
This document describes version 1 of
the Sender Policy Framework (SPF) protocol, whereby ADministrative
Management Domains (ADMDs) can explicitly authorize the hosts that
are allowed to use their domain names, and a receiving host can check
such authorization.
~~~~~~~~

This is completely false. The MAIL FROM identity is not a header. It is an SMTP command issued during the SMTP conversation, after the HELO or EHLO command but before DATA. The From: header is part of the message that is sent after the DATA command. They have absolutely nothing to do with each other, and it is the MAIL FROM identity that is used by SPF, while the From: header is irrelevant.

Please stop replying with misinformation. Thanks.

> processing of notifications, not email deliverability

Why do you separate these two?
If Dreamwidth successfully sent out all notifications, but most of them end up in spam folders of the recipients - then the effect is almost the same as not sending these notifications at all, right?

For example, sudden 100x spike in email sending speed -- may look like spam attack from the perspective of spam filters that email service providers (such as Gmail) have.
If it is the case, then Dreamwidth team may choose not to send emails too fast, even if is technically possible (e.g. allow email sending 10x faster than normal, but not 100x faster).

Edited 2019-12-09 15:56 (UTC)

In the codebase we inherited from LJ, a notification is an asynchronous message to the user, and it is not the same thing as an email. Email is just one of the available delivery methods for notifications. The workers

mark was talking about were dealing with notifications, not email, so that's what he said and I have repeated.

The code is available if you're interested, and I think there are some wiki entries about how the various parts work, including notifications ("ESN" might be the term to search for). However, SES isn't really one of those parts; it and any of its necessary rate-limiting controls are implemented by Amazon.

> mark was talking about were dealing with notifications, not email

Thanks, I see the difference now.

For me, as a Dreamwidth user, the only notification that practically matters is email.
Is it different for most other Dreamwidth users?

ESN wiki lists 3 notification types:
~~~~~~~~~~~
https://wiki.dreamwidth.net/wiki/index.php/ESN
Email
Inbox
DebugLog
~~~~~~~~~~~
but it is not clear how important "Inbox" and "DebugLog" notification types are.

Flat | Top-Level Comments Only

Notifications slow -- but recovering

Re: Amazon SES - misconfigured SPF record?

Re: Amazon SES - misconfigured SPF record?

Re: Amazon SES - misconfigured SPF record?

Re: Amazon SES - misconfigured SPF record?

Re: Amazon SES - misconfigured SPF record?

Dreamwidth.org SPF record

Re: Dreamwidth.org SPF record

Re: Dreamwidth.org SPF record

Re: Dreamwidth.org SPF record

Re: Dreamwidth.org SPF record

Re: Dreamwidth.org SPF record

Re: Dreamwidth.org SPF record

Re: Dreamwidth.org SPF record

Processing notifications and email deliverability

Re: Processing notifications and email deliverability

Notification types