denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
Denise ([staff profile] denise) wrote in [site community profile] dw_maintenance2016-04-15 07:11 pm

(no subject)

People getting mail at Gmail are reporting email delays of about an hour. As far as I can figure out, it's because some of our recent changes to our network have wiped out our existing sending reputation with them and made us start building reputation again, and they're refusing all mail on first delivery and making us re-send it. (It's a common spam reduction technique, because spammers don't bother retrying if the first attempt fails.)

There isn't much we can do about it but wait it out until Gmail decides that we're legit senders again, but we'll poke at it and see if there's anything we can do to make the process go faster. (I doubt there will be, though; Gmail is persnickety.) In the meantime, to get comment notification email faster, you can switch your confirmed email to a different provider, or just refresh your on-DW inbox.

EDIT: And people are now letting me know that mail's delayed to other providers, too, which is probably follow-on effects from having to send everything to Gmail at least twice. There isn't a lot we can do about it; I'm sorry about the hassle, folks.
marahmarie: Sheep go to heaven, goats go to hell (Default)

Re: Oh, wow

[personal profile] marahmarie 2016-04-17 08:17 am (UTC)(link)
Yes, I agree...the same way you feel email is a broken protocol I feel (even more strongly that) HTTP is a broken protocol, and I can get really livid about it, so I hear you, believe me.

And yes, I took it as a a given that anyone might respond that the cost to users and email providers in terms of damage done by opening and clicking infected links in spam emails outweighs any benefit gained by not taking such drastic measures against it. Filtering to the Spam folder could be a great way to solve a few problems around this if the public was more educated and aware about how important it is to be careful but I guess as a practical matter that's never going to happen.

But I feel simple communication is not being utilized effectively enough to prevent a lot of the problems websites like your own can wind up having. I keep running the "friend at Google" scenario through my mind, in which you update DW's MX records, then notice an email slowdown and trace the cause of the delay back to the MX records update, but then you just contact a friend or person serving in some helpful official capacity at Google and bam! the problem just goes away. As it should!

Maybe this is not how the world works but it seems major email providers could have a list of safe senders with websites like your own included and when there's a problem like the MX records update, before they let that slow everything down there's a protocol in place where either they know to contact you or another known site admin to make sure everything is OK before slowing DW's email down or where you know you can get in touch with someone on their side to the same effect.

It's the turnkey-ness of the Internet that gets to me; you can look up someone's profile in seconds online even if you don't know them but major websites and email providers can't communicate with each other more effectively and I just don't understand why.

I mean, I'd bet there are people at Google right now who know what Dreamwidth's about, who can safely assume DW could not possibly be a spam sender unless it got hacked, who know Dreamwidth's like LiveJournal, who know sites like LJ are generally considered safe senders, so there is a shortcut way out of this through better communication between email providers and DW itself, it's just the providers (apparently) don't currently allow that communication to happen, relying on automation instead to (incorrectly) determine and (fail to) solve problems. Which is slowing up things on Dreamwidth for all of us and seems unnecessary.

As to email being broken, while I'm not quite as passionate about why that's bad as you are, I think it's bad enough that it should all be replaced with something like Slack. :)
Edited (typos, clarity) 2016-04-17 08:31 (UTC)
azurelunatic: A glittery black pin badge with a blue holographic star in the middle. (Default)

a friend at Google

[personal profile] azurelunatic 2016-04-18 03:19 am (UTC)(link)

It turns out that it's really hard to find out how much email traffic goes through gmail every day; at least, I wasn't able to find out directly.

According to http://www.radicati.com/wp/wp-content/uploads/2015/02/Email-Statistics-Report-2015-2019-Executive-Summary.pdf (stats for 2015, and predictions through 2019), there are over 205 billion emails exchanged per day.

According to https://emailclientmarketshare.com/ , gmail as a client has a market share of 16%, which doesn't cover the number of people using a desktop app or mobile device to access their email.

So we can guess that gmail is handling around 32, 33 billion emails per day.

I don't know what Dreamwidth's email volume is actually like, but http://www.dreamwidth.org/stats says that currently 4063 accounts have posted at least one entry in the last 24 hours. Let's posit that each of these journals has posted an average of 2 entries. I don't know what average comment stats are like, but I currently have roughly 30,000 entries in my journal, and 58,000 comments on those entries. I'd say that my comments are moderately active -- there are journals with less chatty readers, and roleplaying and socialization communities with far far more. But let's say that 2 comments per entry is average. Since I'm a paid user, I have signed up for a notification of every comment I send (as well as the notification that most people get when someone replies), and it's possible for people to subscribe to threads they're not a part of, so let's say that every comment left sends about 4 notifications. (Again, I don't know the real stats.)

So let's say: 9000 entries (rounding up), each generating 8 notifications. That's 72,000.

72 thousand is 0.00000218181 of 33 billion.

So at a guess, Dreamwidth is not such a significant blip on gmail's radar that a human being would ever be prompted to examine Dreamwidth's trustworthiness status, much less spend the time to select a form letter about changes to MX records and send it off to the address listed as the domain's administrative contact and review the answer when it comes back. (Assuming California minimum wage, that's about $0.50 for a 3-minute process.)

From my experience living and working in Silicon Valley, the "I know a person at $COMPANY" thing is calling in a moderate to major favor. This is what you do if you've already tried the official processes and there's genuinely no way forward, or if the official route leaves you dead in the water for a week or two. I don't have access to Dreamwidth's email stats, but from some of the patterns of grumbling, Gmail pitches some sort of woebucket over Dreamwidth's notifications maybe 4+ times a year.

Dreamwidth staff and volunteers do not, as a rule, help Google employees move house quite that often.
marahmarie: Sheep go to heaven, goats go to hell (Default)

Re: a friend at Google

[personal profile] marahmarie 2016-04-19 04:27 am (UTC)(link)
Well, that does make DW seem like a tiny speck on Google's horizon. Thanks for digging up so many stats and doing so much math on that; those are some truly eye-opening results.

*is sad for our relative speckiness*

Where our thoughts kind of diverge is where you write, "So at a guess, Dreamwidth is not such a significant blip on gmail's radar that a human being would ever be prompted to examine Dreamwidth's trustworthiness status", because the way I see a possible answer to this is in the fact that it's not how big or small Dreamwidth is, it's how much can we trust the reputation, longevity, and current visible/knowable behavior of any website, regardless of its size.

If....if there was something like a cross between Google's rankings for websites (which more and more do seem to sift for trust, reputation and reliability factors, overall) and Web of Trust, but for email servers...I'm just not sure exactly how it would be done as far as nuts and bolts go but I'm thinking that unlike WOT it could be mostly automated/algorithm driven and then checked over by a WOT-like team, only perhaps a paid team with volunteers as opposed to the entire (unpaid) Internet that WOT draws from now to rate All The Websites.

And from this vague and fuzzy idea I'm having that's sort of hurting my brain to flesh out, a whitelist that stays updated in real time can be created, perhaps distributed to major email providers, and any checking against it for safe or unsafe senders, say when some website's MX records suddenly change, can also be automated, requiring followup only when a) the website is having other problems or b) other obvious red flags become apparent.

As to the friend at Google, it pains me to show my age by recalling the web as a much smaller place than it is. I had (to this day a completely unknown) friend at Wordpress who kind of helped me out back in the day (without me asking!) when even Wordpress wasn't too big (but it was getting pretty big!). Of course as a user of the site, blah blah blah...(I guess he or she was a a fan, but as they chose to remain anonymous, I'll probably never know exactly what inspired them).

So, there might be billions of websites of every stripe and flavor but there are only so many social media sites with longevity, good reputation and loyal users; even if Dreamwidth ranks at the low end of userbase among them, it surely must rank (actually, I'd be curious to know where on the scale of most-used social media sites it stands, but my gut feeling is it's at least pretty well known). Silicon Valley et al pays attention to social media and knows who's who, so it just rankles me a bit that communication can't be better than it is.
Edited (typo) 2016-04-19 04:31 (UTC)
azurelunatic: A castle with rockets and fire cannons with the DW D on it. (Castle Dreamwidth)

Re: a friend at Google

[personal profile] azurelunatic 2016-04-19 04:40 am (UTC)(link)
I have the opportunity to say where my Open Source involvement is, at work and at tech meetup events. When I say "Dreamwidth" I mostly get blank looks; maybe 1 time out of 100 or 200 I get a delighted "ooo Dreamwidth!" or a "Oh huh, are they still around?"; when I say "a code fork of LiveJournal" I get variations on "Man, I used to have a LiveJournal..." about 95% of the time. Sometimes they recognize Dreamwidth if I mention that it's one of the few open source projects which are majority women. So off the cuff I would not say that it is well known outside of certain fannish circles, which do not line up with Silly Valley dev circles in the way they may have some time ago.
marahmarie: Sheep go to heaven, goats go to hell (Default)

Re: a friend at Google

[personal profile] marahmarie 2016-04-19 04:58 am (UTC)(link)
or a "Oh huh, are they still around?"

That's actually what I'm often tempted to say about, you know, the other half of that equation. Another of my gut feelings is LJ has been picking up steam lately and good for them, as I feel (with the cross-posting abilities we have in place that I see a lot of fairly popular DW users taking advantage of each day) that it might actually help Dreamwidth to grow, in the long run.

I can't really speak to how SV's great teaming masses react when DW's name gets dropped...when I say my gut feeling is DW is at least well known, I'm referring more to the upper echelon of Valleyites, whose job it is to kind of stay well-informed (at least I would think so) not so much their dev circles (though I have mad respect for some of them and think they might have at least some knowledge of us, too).
Edited (typos, incomplete sentence) 2016-04-19 05:22 (UTC)