dw_maintenance | (no subject)

(no subject)

People getting mail at Gmail are reporting email delays of about an hour. As far as I can figure out, it's because some of our recent changes to our network have wiped out our existing sending reputation with them and made us start building reputation again, and they're refusing all mail on first delivery and making us re-send it. (It's a common spam reduction technique, because spammers don't bother retrying if the first attempt fails.)

There isn't much we can do about it but wait it out until Gmail decides that we're legit senders again, but we'll poke at it and see if there's anything we can do to make the process go faster. (I doubt there will be, though; Gmail is persnickety.) In the meantime, to get comment notification email faster, you can switch your confirmed email to a different provider, or just refresh your on-DW inbox.

EDIT: And people are now letting me know that mail's delayed to other providers, too, which is probably follow-on effects from having to send everything to Gmail at least twice. There isn't a lot we can do about it; I'm sorry about the hassle, folks.

Flat | Top-Level Comments Only

People getting mail at Gmail are reporting email delays of about an hour.

For reasons related to Microsoft flagging DW's email as spam, I forward my DW email from Gmail to Outlook, which means the same delays, I guess, and that I have no provider left to switch to that I could tolerate (in fact, I can't tolerate GMail nor Yahoo, which is why I forward email to Outlook - the latter's interfaces are too garish, cluttered and arcane for me to deal with).

But I'm wondering how Dreamwidth is not whitelisted with all major email providers by now (it's been online for eight years! I think!). It seems weird that major email providers don't make more of an effort to ensure legitimate organizations such as Dreamwidth don't get flagged as spam senders.

And it's frustrating. If the bottlenecks continue with big providers as reported in the OP I guess I'll have to find a smaller provider to switch to (or buy a website to set up an email server like I've talked about but have never actually done) as Outlook is my favorite but has not, as far as I know, let DW's email through in the last year (edit: just checked my own DW and according to a post I made about it, it's been a year and four months since that particular problem started).

Edited (clarity) 2016-04-16 05:57 (UTC)

My buddy and I were just chatting tonight about spam blacklists and how hard it is to navigate even when you know what you're doing. He manages his own email server and subscribes to one of the less extortionate blacklists (there are many spam blacklists out there, and some are more reliable than others) and this week he got a call from his insurance agent asking whether he had the information ready to send over that the agent had asked for in his last email. Turns out that while the agency's automatic mass email stuff was getting through just fine, his agent was sending from home or some such thing, via an outgoing server shared with a lot of other customers ... and when one of the 1000+ other people using that server gets compromised and starts sending out spam, then everything sent via that outgoing server is suspect. And it had got onto the blacklist, badly enough that my buddy's mail server wasn't just dropping it in the spam bucket, it was just kinda dropping it without acknowledgment. He would have never known if the agent hadn't called, because other email from the same domain (but being sent via a different mail server) was getting through. Best of luck if you do wind up managing your own server! I salute everyone who makes the attempt, because the world needs people willing to make a go of it to see if they can.

Longevity doesn't necessarily mean that much when there's been a sudden change in pattern, since there's always the chance that a spammer compromised a legitimate provider and is now attempting to fire the spam-cannon double-quick before they get shut down and have to compromise someone else's mail server. I suspect that the bigger the provider, the less they care about individual email getting through as long as they can lower their spam volume and reduce their hours of labor to email traffic ratio. :\

So basically I'd have to buy my own rack space (not just my own space on, say, a shared web server) to ensure I don't pay for the misdeeds done by others on my rack. I should have known there was a catch!

It might be a bit - controversial - to say so but the second I read the OP I wished email providers had not gotten so "good" at flagging and filtering spam (I use the word "good" loosely because while I'm sure that compared to maybe 10 years they catch some bigger, more obvious miscreants now that otherwise would have basically flooded out email servers and inundated customers with spam) because they suck at it, the proof being moments exactly like what Dreamwidth is going through now with GMail.

I think the good get punished with the bad and that that level of strictness is unnecessary when that's what your email account's Spam folder is for, and every email account everywhere has one, so why not just concentrate their endeavors on getting better at filtering into it instead of basically knocking the knees out from under so many other legitimate website's functionality?

Your buddy's story kind of highlights what I mean - why not reroute newly suspected spam to the Spam folder instead of into The Big Black Void? Who does that help? If indeed the email server was taken over nefariously wouldn't it be better (for your buddy's email provider, in the long run, if not necessarily for other customers in the short run) if your buddy could see something was amiss by the strange emails getting sent from that address and be able to report the problem himself? An argument could be made that people don't check their Spam folders often or carefully enough to catch such things but I think the success rate that way would still be higher than zero, which is the success rate they'll have picking the Big Black Void over anyone's inbox.

Edited 2016-04-17 04:26 (UTC)

It costs millions of dollars annually to store and transmit spam; I've seen estimates that up to 80% of email volume is spam. It's also the #1 malware vector resulting in billions of dollars of damage, cleanup, and lost productivity a year. "Just pass everything through to people's spam filters" really isn't practical, even though the countermeasures are a fucking pain in the ass to deal with.

Email is a fundamentally broken protocol, basically, and 30 years of bolting fixes on top of it hasn't done anything but make it a fundamentally broken protocol with lots of annoying things on top.

Yes, I agree...the same way you feel email is a broken protocol I feel (even more strongly that) HTTP is a broken protocol, and I can get really livid about it, so I hear you, believe me.

And yes, I took it as a a given that anyone might respond that the cost to users and email providers in terms of damage done by opening and clicking infected links in spam emails outweighs any benefit gained by not taking such drastic measures against it. Filtering to the Spam folder could be a great way to solve a few problems around this if the public was more educated and aware about how important it is to be careful but I guess as a practical matter that's never going to happen.

But I feel simple communication is not being utilized effectively enough to prevent a lot of the problems websites like your own can wind up having. I keep running the "friend at Google" scenario through my mind, in which you update DW's MX records, then notice an email slowdown and trace the cause of the delay back to the MX records update, but then you just contact a friend or person serving in some helpful official capacity at Google and bam! the problem just goes away. As it should!

Maybe this is not how the world works but it seems major email providers could have a list of safe senders with websites like your own included and when there's a problem like the MX records update, before they let that slow everything down there's a protocol in place where either they know to contact you or another known site admin to make sure everything is OK before slowing DW's email down or where you know you can get in touch with someone on their side to the same effect.

It's the turnkey-ness of the Internet that gets to me; you can look up someone's profile in seconds online even if you don't know them but major websites and email providers can't communicate with each other more effectively and I just don't understand why.

I mean, I'd bet there are people at Google right now who know what Dreamwidth's about, who can safely assume DW could not possibly be a spam sender unless it got hacked, who know Dreamwidth's like LiveJournal, who know sites like LJ are generally considered safe senders, so there is a shortcut way out of this through better communication between email providers and DW itself, it's just the providers (apparently) don't currently allow that communication to happen, relying on automation instead to (incorrectly) determine and (fail to) solve problems. Which is slowing up things on Dreamwidth for all of us and seems unnecessary.

As to email being broken, while I'm not quite as passionate about why that's bad as you are, I think it's bad enough that it should all be replaced with something like Slack. :)

Edited (typos, clarity) 2016-04-17 08:31 (UTC)

It turns out that it's really hard to find out how much email traffic goes through gmail every day; at least, I wasn't able to find out directly.

According to http://www.radicati.com/wp/wp-content/uploads/2015/02/Email-Statistics-Report-2015-2019-Executive-Summary.pdf (stats for 2015, and predictions through 2019), there are over 205 billion emails exchanged per day.

According to https://emailclientmarketshare.com/ , gmail as a client has a market share of 16%, which doesn't cover the number of people using a desktop app or mobile device to access their email.

So we can guess that gmail is handling around 32, 33 billion emails per day.

I don't know what Dreamwidth's email volume is actually like, but http://www.dreamwidth.org/stats says that currently 4063 accounts have posted at least one entry in the last 24 hours. Let's posit that each of these journals has posted an average of 2 entries. I don't know what average comment stats are like, but I currently have roughly 30,000 entries in my journal, and 58,000 comments on those entries. I'd say that my comments are moderately active -- there are journals with less chatty readers, and roleplaying and socialization communities with far far more. But let's say that 2 comments per entry is average. Since I'm a paid user, I have signed up for a notification of every comment I send (as well as the notification that most people get when someone replies), and it's possible for people to subscribe to threads they're not a part of, so let's say that every comment left sends about 4 notifications. (Again, I don't know the real stats.)

So let's say: 9000 entries (rounding up), each generating 8 notifications. That's 72,000.

72 thousand is 0.00000218181 of 33 billion.

So at a guess, Dreamwidth is not such a significant blip on gmail's radar that a human being would ever be prompted to examine Dreamwidth's trustworthiness status, much less spend the time to select a form letter about changes to MX records and send it off to the address listed as the domain's administrative contact and review the answer when it comes back. (Assuming California minimum wage, that's about $0.50 for a 3-minute process.)

From my experience living and working in Silicon Valley, the "I know a person at $COMPANY" thing is calling in a moderate to major favor. This is what you do if you've already tried the official processes and there's genuinely no way forward, or if the official route leaves you dead in the water for a week or two. I don't have access to Dreamwidth's email stats, but from some of the patterns of grumbling, Gmail pitches some sort of woebucket over Dreamwidth's notifications maybe 4+ times a year.

Dreamwidth staff and volunteers do not, as a rule, help Google employees move house quite that often.

Well, that does make DW seem like a tiny speck on Google's horizon. Thanks for digging up so many stats and doing so much math on that; those are some truly eye-opening results.

*is sad for our relative speckiness*

Where our thoughts kind of diverge is where you write, "So at a guess, Dreamwidth is not such a significant blip on gmail's radar that a human being would ever be prompted to examine Dreamwidth's trustworthiness status", because the way I see a possible answer to this is in the fact that it's not how big or small Dreamwidth is, it's how much can we trust the reputation, longevity, and current visible/knowable behavior of any website, regardless of its size.

If....if there was something like a cross between Google's rankings for websites (which more and more do seem to sift for trust, reputation and reliability factors, overall) and Web of Trust, but for email servers...I'm just not sure exactly how it would be done as far as nuts and bolts go but I'm thinking that unlike WOT it could be mostly automated/algorithm driven and then checked over by a WOT-like team, only perhaps a paid team with volunteers as opposed to the entire (unpaid) Internet that WOT draws from now to rate All The Websites.

And from this vague and fuzzy idea I'm having that's sort of hurting my brain to flesh out, a whitelist that stays updated in real time can be created, perhaps distributed to major email providers, and any checking against it for safe or unsafe senders, say when some website's MX records suddenly change, can also be automated, requiring followup only when a) the website is having other problems or b) other obvious red flags become apparent.

As to the friend at Google, it pains me to show my age by recalling the web as a much smaller place than it is. I had (to this day a completely unknown) friend at Wordpress who kind of helped me out back in the day (without me asking!) when even Wordpress wasn't too big (but it was getting pretty big!). Of course as a user of the site, blah blah blah...(I guess he or she was a a fan, but as they chose to remain anonymous, I'll probably never know exactly what inspired them).

So, there might be billions of websites of every stripe and flavor but there are only so many social media sites with longevity, good reputation and loyal users; even if Dreamwidth ranks at the low end of userbase among them, it surely must rank (actually, I'd be curious to know where on the scale of most-used social media sites it stands, but my gut feeling is it's at least pretty well known). Silicon Valley et al pays attention to social media and knows who's who, so it just rankles me a bit that communication can't be better than it is.

Edited (typo) 2016-04-19 04:31 (UTC)

I have the opportunity to say where my Open Source involvement is, at work and at tech meetup events. When I say "Dreamwidth" I mostly get blank looks; maybe 1 time out of 100 or 200 I get a delighted "ooo Dreamwidth!" or a "Oh huh, are they still around?"; when I say "a code fork of LiveJournal" I get variations on "Man, I used to have a LiveJournal..." about 95% of the time. Sometimes they recognize Dreamwidth if I mention that it's one of the few open source projects which are majority women. So off the cuff I would not say that it is well known outside of certain fannish circles, which do not line up with Silly Valley dev circles in the way they may have some time ago.

or a "Oh huh, are they still around?"

That's actually what I'm often tempted to say about, you know, the other half of that equation. Another of my gut feelings is LJ has been picking up steam lately and good for them, as I feel (with the cross-posting abilities we have in place that I see a lot of fairly popular DW users taking advantage of each day) that it might actually help Dreamwidth to grow, in the long run.

I can't really speak to how SV's great teaming masses react when DW's name gets dropped...when I say my gut feeling is DW is at least well known, I'm referring more to the upper echelon of Valleyites, whose job it is to kind of stay well-informed (at least I would think so) not so much their dev circles (though I have mad respect for some of them and think they might have at least some knowledge of us, too).

Edited (typos, incomplete sentence) 2016-04-19 05:22 (UTC)

Given the volume we're discussing, whitelists are hella impractical for this sort of thing, alas. There are a number of protocols that have been bolted on top of email that providers can use to cause a receiver to identify their mail as more likely to be legit. They're extremely bolted-on, massively imperfect, and each of them can break various things a mail sender might want to do in various ways. We use most of them -- the ones that don't break our particular use case -- but since they're extremely bolted-on and massively imperfect, major email providers don't always put a ton of weight on them.

The fundamental, basic problem that email is a protocol that assumes trust and good faith, deriving as it does from the days when everybody knew everybody who was on the internet and if somebody was doing a dumbshit thing you could call up the network admin at their institution and ask them to walk down the hall and tell their user to stop doing dumbshit things, and therefore doesn't scale at all to tens of billions of emails being sent daily. Major email providers like Gmail have zero incentive to ensure perfect delivery and significant incentive to reduce the amount of spam they pass through and store.

At the least a dedicated IP; I'm a little fuzzy on how easy it would be to tell that two things were on the same physical box as opposed to just the same data center if they have separate IPs; that's a level of forensics a little beyond my expertise.

Even though my buddy's running his own mail server and gets to make his own rules, the straight-up blackholing of suspected spammers is helpful because it's basically a "woops, there's no mail server there anymore" message, rather than a "there's a mail server here but it says Talk to the Hand b/c the inbox ain't listening" (an explicit hard bounce message) or a "Heyyyyyyy we got your email bro!" (accept and send to spambox). Spammers who get a "heyyyyyyyyyyy we got your email bro!" will increase the amount of spam they spray at that server, since it's just indicated it's a possible target. So he's made that choice deliberately and from an informed place. Unfortunately this time it did in fact catch legit mail. So he sent the relevant information along to the agent so the agent could give it to his IT guy to go talk to the blacklist maintainers and get un-blacklisted.

Edited 2016-04-17 08:30 (UTC)

I failed to take into account that letting spam through encourages spammers to simply send more spam. Thanks for explaining that as it illuminates a lot (admittedly, my ideas on how spam could be handled are likely naive and uninformed, which is why I ask a lot of questions and wonder openly about a lot of things - there are probably good reasons my ideas are impractical or just plain undoable but I can't envision what they are without looking a lot more into the topic).

Yup. Encouraging spammers: feeding seagulls is a better life choice!

Never, ever feed the seagulls! If you practically grew up on the beach like I did then you know this is blasphemy! (Spent major chunks of my childhood/life on one or another.)

Barely any worse than feeding spammers, though, I must admit.

I just had to say that the phrasing of your examples was hysterical. :D Still making me giggle.

My excessively helpful brain has now started to ponder life as it would be if email servers had names and personalities like Culture ships.

I think Denise said something about this in the OP -- something about their most recent code push has basically made them look like a new email sender, so they were good and whitelisted before but now they're starting from scratch. It'll probably not take long to get whitelisted again, though.

We changed some of our MX records, and changes to your MX can make receivers recalculate your reputation -- to Gmail right now, we look like we just started sending email, so we have to build the reputation back up.

I've been running my own mail server for ... *counts on fingers* oh God fifteen years now. It's definitely good for not having to deal with a provider's fuckery, but it's a lot more trouble to run mail than it is to run a webserver (which is pretty much start-and-forget, past initial configuration). I've considered outsourcing the mail end of things more times than I can count.

We changed some of our MX records

I was curious what that looks like (I last set up MX records 10 years ago on my own website, so my memory is long gone on that) so I googled some and found a checking service which says DW has only one MX record - http://www.intodns.com/dreamwidth.org. I'm not sure how to interpret the data as the website checking DW's information could of course be wrong, or DW could in fact have just one MX record, like it says. I'm passing along what it says just in case it in any way relates to the current problem, which of course I doubt, but just to be on the safe side...

And thanks for the warning on running one's own mailserver...if it's that much trouble I probably won't do it (and this might make people who know me laugh, but I was eying the comment up above about AOL not experiencing delays with a lot of hope...because as much as I don't like AOL, as a company, I think I *would* switch to their interface before I'd switch to using Yahoo's or Gmail's).

Azz was telling me a story above that kind of reflects what DW's going through now with GMail, only worse (mail was not slow - nope, it simply disappeared thanks to someone else's bad behavior on the same server!) which makes me realize I'd probably have to buy my own rack space just to avoid most of the possible issues. With running your own mailserver being decidedly more difficult than running your own web server, I see no reason to put myself through that (after all, dedicated rackspace is very expensive).

I hope all the email slowness gets resolved soon!

Edited (typo) 2016-04-17 05:29 (UTC)

*raises hand* So I work in IT Operations for a Large Tech Company and I used to work in the group that helped manage our email, and there is just so much of it that it's really hard. Whitelisting only works for specific IP addresses or addresses; if those keep changing, or if the protocol used for them changes, the whitelist is no longer accurate and you have to update it. This is actually an issue we've had at work a few times, where the ops guys think something is whitelisted and it actually isn't and then things break.

Most email providers on the scale of Outlook or Gmail or whoever do have a process for getting yourself unblocked, but it's generally really opaque and there's no guarantee the results will last (as DW has seen).

Like you, I forward my gmail notifications to my Outlook account, so this just adds more delay but I'm out of providers and have no wish to set up a new account...

Yeah. The email delays we're seeing are not fun, though. I checked how long last night as Azz and I were bouncing back and forth on this page and the delays were so long I was only able to see their replies when a) I couldn't edit my comment about whitelisting (as it had already been replied to) or b) I refreshed the page. So at the least 2-3 hours (I gave up trying to see how long it took to hit my Inbox after that).

On the face of it, email is pretty much a whitelist by default, as Denise says, so right now we deal with the problems that so easily causes by blacklisting (or otherwise kind of torturing) suspect senders. Spammers expect to be blacklisted so probably know enough to realize they need space at multiple servers/locations. I'm sure the more proficient among them just plan accordingly.

So blacklisting is not an incentive to not spam; it's an incentive to own more IP blocks/domain names/servers. Unless spammers run their own servers or use botnets (which I'm sure they do in addition to merely spamming from, say, their own publicly available websites) the only people incentivized to support blacklists are the ones selling spammers server space and domain names. Because spammers need a lot of both.

I'm thinking since email is basically whitelist by default and since blacklists can often be - I hate them, quite frankly I think they're bullshit - inaccurate, to say the least, not to say far from complete by the very nature of how spammers work tirelessly to circumvent them, that working more with the whitelist idea might be better in the long run.

Incentivize that; make it so everyone who sends email realizes they get priority treatment if they behave, but simply get vaporized (in the long run) if they don't (but if they do behave badly, then as soon as they correct their behavior send their email sailing right through again; nothing is permanent in this system because it simply reacts to how you behave, and does so as close to real time as technology will allow). No more blacklists outside of a record of which senders you vaporize. Automated algorithms pick apart the good from the bad and make the system impossible to game by flagging for enough signals to ensure the sender is legit.

If a legit sender such as Dreamwidth changes, say, a MX record and that throws a red flag then until it can be dealt with (again, perhaps in combination with the currently non-existent customer service we have so far) then rather than initially slow the email down through forced resends or outright disappearing it right off the bat, the email is linked to the same way a webpage can be.

So in your Inbox? You click a link to open the email and all you get is another link; when you hover that you basically have to sign a clickwrap agreement saying you realize the mail in question might destroy your computer, your life and steal you or your neighbor's firstborn and everyone's dog before you can even download it. Even then, images and links within the suspect email stay blocked with more warnings you have to agree to before you can open them.

This might solve some of the server space problems with storing spam Denise mentioned, as nothing suspect is shown to the user except the email title - not without extensive warning to the user on every suspect email and a very explicit agreement being clicked upon that the user does in fact understand things could go really, really bad. If the user doesn't try to open such emails within, say, two weeks, they're flushed from the server because reaching people's eyeballs through their inboxes is a privilege, not a right.

Besides that, just incentivize good behavior to make people want to remain on a whitelist and maybe force at least some of the less determined spammers to clean up their acts, so say, in return for good behavior, prioritize all their email delivery; over time as trust is built give them little Safe Sender icons like you see in Outlook for - I think it's eBay and Microsoft's own mail - to show the user whom to trust; offer reasonable paid subscription services that provide instant access to email tech support to - why aren't things like this done now?

I'm not sure why email seems to lack salespeople with some positive spin on things but it needs a way to unite the backend work of delivering safe email with the front end work of encouraging people to want to do exactly that. I can't be the first or the 10 millionth person to think so and most of my ideas are probably not original, it's just a remix of some I've seen and had over time.

Again, this is all a bit pie in the sky on my part and there are probably many reasons why what I'm suggesting can't work as one cohesive, completely secure program, but that's where I realize more minds on the same problem have either already discovered it can't be done or maybe just haven't iterated or reiterated it enough yet.

Edited (more info) 2016-04-20 03:44 (UTC)

Flat | Top-Level Comments Only

(no subject)

Oh, wow

Re: Oh, wow

Re: Oh, wow

Re: Oh, wow

Re: Oh, wow

a friend at Google

Re: a friend at Google

Re: a friend at Google

Re: a friend at Google

Re: a friend at Google

Re: Oh, wow

Re: Oh, wow

Re: Oh, wow

Re: Oh, wow

Re: Oh, wow

Re: Oh, wow

Re: Oh, wow

Re: Oh, wow

Re: Oh, wow

Re: Oh, wow

Re: Oh, wow