dw_maintenance | (no subject)

People getting mail at Gmail are reporting email delays of about an hour. As far as I can figure out, it's because some of our recent changes to our network have wiped out our existing sending reputation with them and made us start building reputation again, and they're refusing all mail on first delivery and making us re-send it. (It's a common spam reduction technique, because spammers don't bother retrying if the first attempt fails.)

There isn't much we can do about it but wait it out until Gmail decides that we're legit senders again, but we'll poke at it and see if there's anything we can do to make the process go faster. (I doubt there will be, though; Gmail is persnickety.) In the meantime, to get comment notification email faster, you can switch your confirmed email to a different provider, or just refresh your on-DW inbox.

EDIT: And people are now letting me know that mail's delayed to other providers, too, which is probably follow-on effects from having to send everything to Gmail at least twice. There isn't a lot we can do about it; I'm sorry about the hassle, folks.

Flat | Top-Level Comments Only

It turns out that it's really hard to find out how much email traffic goes through gmail every day; at least, I wasn't able to find out directly.

According to http://www.radicati.com/wp/wp-content/uploads/2015/02/Email-Statistics-Report-2015-2019-Executive-Summary.pdf (stats for 2015, and predictions through 2019), there are over 205 billion emails exchanged per day.

According to https://emailclientmarketshare.com/ , gmail as a client has a market share of 16%, which doesn't cover the number of people using a desktop app or mobile device to access their email.

So we can guess that gmail is handling around 32, 33 billion emails per day.

I don't know what Dreamwidth's email volume is actually like, but http://www.dreamwidth.org/stats says that currently 4063 accounts have posted at least one entry in the last 24 hours. Let's posit that each of these journals has posted an average of 2 entries. I don't know what average comment stats are like, but I currently have roughly 30,000 entries in my journal, and 58,000 comments on those entries. I'd say that my comments are moderately active -- there are journals with less chatty readers, and roleplaying and socialization communities with far far more. But let's say that 2 comments per entry is average. Since I'm a paid user, I have signed up for a notification of every comment I send (as well as the notification that most people get when someone replies), and it's possible for people to subscribe to threads they're not a part of, so let's say that every comment left sends about 4 notifications. (Again, I don't know the real stats.)

So let's say: 9000 entries (rounding up), each generating 8 notifications. That's 72,000.

72 thousand is 0.00000218181 of 33 billion.

So at a guess, Dreamwidth is not such a significant blip on gmail's radar that a human being would ever be prompted to examine Dreamwidth's trustworthiness status, much less spend the time to select a form letter about changes to MX records and send it off to the address listed as the domain's administrative contact and review the answer when it comes back. (Assuming California minimum wage, that's about $0.50 for a 3-minute process.)

From my experience living and working in Silicon Valley, the "I know a person at $COMPANY" thing is calling in a moderate to major favor. This is what you do if you've already tried the official processes and there's genuinely no way forward, or if the official route leaves you dead in the water for a week or two. I don't have access to Dreamwidth's email stats, but from some of the patterns of grumbling, Gmail pitches some sort of woebucket over Dreamwidth's notifications maybe 4+ times a year.

Dreamwidth staff and volunteers do not, as a rule, help Google employees move house quite that often.

Well, that does make DW seem like a tiny speck on Google's horizon. Thanks for digging up so many stats and doing so much math on that; those are some truly eye-opening results.

*is sad for our relative speckiness*

Where our thoughts kind of diverge is where you write, "So at a guess, Dreamwidth is not such a significant blip on gmail's radar that a human being would ever be prompted to examine Dreamwidth's trustworthiness status", because the way I see a possible answer to this is in the fact that it's not how big or small Dreamwidth is, it's how much can we trust the reputation, longevity, and current visible/knowable behavior of any website, regardless of its size.

If....if there was something like a cross between Google's rankings for websites (which more and more do seem to sift for trust, reputation and reliability factors, overall) and Web of Trust, but for email servers...I'm just not sure exactly how it would be done as far as nuts and bolts go but I'm thinking that unlike WOT it could be mostly automated/algorithm driven and then checked over by a WOT-like team, only perhaps a paid team with volunteers as opposed to the entire (unpaid) Internet that WOT draws from now to rate All The Websites.

And from this vague and fuzzy idea I'm having that's sort of hurting my brain to flesh out, a whitelist that stays updated in real time can be created, perhaps distributed to major email providers, and any checking against it for safe or unsafe senders, say when some website's MX records suddenly change, can also be automated, requiring followup only when a) the website is having other problems or b) other obvious red flags become apparent.

As to the friend at Google, it pains me to show my age by recalling the web as a much smaller place than it is. I had (to this day a completely unknown) friend at Wordpress who kind of helped me out back in the day (without me asking!) when even Wordpress wasn't too big (but it was getting pretty big!). Of course as a user of the site, blah blah blah...(I guess he or she was a a fan, but as they chose to remain anonymous, I'll probably never know exactly what inspired them).

So, there might be billions of websites of every stripe and flavor but there are only so many social media sites with longevity, good reputation and loyal users; even if Dreamwidth ranks at the low end of userbase among them, it surely must rank (actually, I'd be curious to know where on the scale of most-used social media sites it stands, but my gut feeling is it's at least pretty well known). Silicon Valley et al pays attention to social media and knows who's who, so it just rankles me a bit that communication can't be better than it is.

Edited (typo) 2016-04-19 04:31 (UTC)

I have the opportunity to say where my Open Source involvement is, at work and at tech meetup events. When I say "Dreamwidth" I mostly get blank looks; maybe 1 time out of 100 or 200 I get a delighted "ooo Dreamwidth!" or a "Oh huh, are they still around?"; when I say "a code fork of LiveJournal" I get variations on "Man, I used to have a LiveJournal..." about 95% of the time. Sometimes they recognize Dreamwidth if I mention that it's one of the few open source projects which are majority women. So off the cuff I would not say that it is well known outside of certain fannish circles, which do not line up with Silly Valley dev circles in the way they may have some time ago.

or a "Oh huh, are they still around?"

That's actually what I'm often tempted to say about, you know, the other half of that equation. Another of my gut feelings is LJ has been picking up steam lately and good for them, as I feel (with the cross-posting abilities we have in place that I see a lot of fairly popular DW users taking advantage of each day) that it might actually help Dreamwidth to grow, in the long run.

I can't really speak to how SV's great teaming masses react when DW's name gets dropped...when I say my gut feeling is DW is at least well known, I'm referring more to the upper echelon of Valleyites, whose job it is to kind of stay well-informed (at least I would think so) not so much their dev circles (though I have mad respect for some of them and think they might have at least some knowledge of us, too).

Edited (typos, incomplete sentence) 2016-04-19 05:22 (UTC)

Given the volume we're discussing, whitelists are hella impractical for this sort of thing, alas. There are a number of protocols that have been bolted on top of email that providers can use to cause a receiver to identify their mail as more likely to be legit. They're extremely bolted-on, massively imperfect, and each of them can break various things a mail sender might want to do in various ways. We use most of them -- the ones that don't break our particular use case -- but since they're extremely bolted-on and massively imperfect, major email providers don't always put a ton of weight on them.

The fundamental, basic problem that email is a protocol that assumes trust and good faith, deriving as it does from the days when everybody knew everybody who was on the internet and if somebody was doing a dumbshit thing you could call up the network admin at their institution and ask them to walk down the hall and tell their user to stop doing dumbshit things, and therefore doesn't scale at all to tens of billions of emails being sent daily. Major email providers like Gmail have zero incentive to ensure perfect delivery and significant incentive to reduce the amount of spam they pass through and store.

(no subject)

a friend at Google

Re: a friend at Google

Re: a friend at Google

Re: a friend at Google

Re: a friend at Google