denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
Denise ([staff profile] denise) wrote in [site community profile] dw_maintenance2016-04-15 07:11 pm

(no subject)

People getting mail at Gmail are reporting email delays of about an hour. As far as I can figure out, it's because some of our recent changes to our network have wiped out our existing sending reputation with them and made us start building reputation again, and they're refusing all mail on first delivery and making us re-send it. (It's a common spam reduction technique, because spammers don't bother retrying if the first attempt fails.)

There isn't much we can do about it but wait it out until Gmail decides that we're legit senders again, but we'll poke at it and see if there's anything we can do to make the process go faster. (I doubt there will be, though; Gmail is persnickety.) In the meantime, to get comment notification email faster, you can switch your confirmed email to a different provider, or just refresh your on-DW inbox.

EDIT: And people are now letting me know that mail's delayed to other providers, too, which is probably follow-on effects from having to send everything to Gmail at least twice. There isn't a lot we can do about it; I'm sorry about the hassle, folks.
alexseanchai: Blue and purple lightning (Default)

[personal profile] alexseanchai 2016-04-15 11:17 pm (UTC)(link)
Thanks!
minim_calibre: (Default)

[personal profile] minim_calibre 2016-04-15 11:26 pm (UTC)(link)
I'm noticing them with Yandex, too, which is odd for Yandex. (I switched to it because I had so many issues with other providers.)

That said, it's just in the last couple of hours that I've seen it.
vorbarra: (Default)

[personal profile] vorbarra 2016-04-15 11:27 pm (UTC)(link)
Thanks for the explanation! I appreciate knowing what's going on. :)
niqaeli: cat with arizona flag in the background (Default)

[personal profile] niqaeli 2016-04-15 11:29 pm (UTC)(link)
Oh, Gmail. *facepalm*

Thanks for the update!
broken_arrow: (Default)

[personal profile] broken_arrow 2016-04-15 11:34 pm (UTC)(link)
I use a different provider, Yahoo Mail, and I'm not getting notifs either. They were delayed by about an hour last night, and tonight I've been tagging with someone for two hours now and haven't gotten a single notif.

Just wanted to let you know, it's not just gmail.
resipiscent: (Default)

[personal profile] resipiscent 2016-04-15 11:37 pm (UTC)(link)
hello! i'm on hotmail and i'm GETTING notifs but they're very delayed! thank you for the update though :)
resipiscent: (Default)

[personal profile] resipiscent 2016-04-16 01:55 am (UTC)(link)
that's cool. i'm just happy that you guys are always right on this stuff!
utselet: (Default)

[personal profile] utselet 2016-04-15 11:39 pm (UTC)(link)
Just to chime in, it's way more than an hour, and has been since last night. I'm getting some notifs in my inbox from hours ago, and I haven't gotten email notifs at all on some replies.
jordannamorgan: Claude Rains as Jack Griffin, "The Invisible Man". (Jack Griffin)

[personal profile] jordannamorgan 2016-04-16 12:34 am (UTC)(link)
Would the backups caused by the Gmail mess be delaying replies to other email services (as in not-free, ISP-based ones)? Because I'm on one of those, and... yep. *sigh*
jordannamorgan: Edward Elric, "Fullmetal Alchemist". (FMA Move Forward)

[personal profile] jordannamorgan 2016-04-16 12:57 am (UTC)(link)
Understood. Knowing it's just a temporary thing to slog through helps. Thanks for the excellent communication!
the: (jeffrey characterwheaties)

[personal profile] the 2016-04-16 12:35 am (UTC)(link)
is there a super loosy-goosy ETA on when notifs might be back to normal?
kore: (Default)

[personal profile] kore 2016-04-16 01:12 am (UTC)(link)
I just got Gmail copies of comments I made at 3:29 pm (comment timestamp), timestamped 5:59 in Gmail, if that helps any. I don't seem to be missing any, though, they're just delayed. For right now I'm just keeping my DW inbox open in its own tab.
borneinblood: (looking out on the world)

[personal profile] borneinblood 2016-04-16 01:33 am (UTC)(link)
Yeah, this is about what it's looking like for me as well, as far as I can tell. I've definitely been checking the DW inbox as best I can in the meanwhile, though.
kore: (Default)

[personal profile] kore 2016-04-16 12:51 am (UTC)(link)
Oh, GMAIL. (They've started to send good mail to my spam and spam to my inbox again.) Also, thank you for keeping us updated and being so informative about issues like this.
brave_heart_verity: (Default)

[personal profile] brave_heart_verity 2016-04-16 01:59 am (UTC)(link)
Try more like three hours. I just got the notification about this post.

Livable, but annoying. Any idea how long it'll take before it gets better?
brave_heart_verity: (Default)

[personal profile] brave_heart_verity 2016-04-16 05:20 am (UTC)(link)
We only hate it when it doesn't do what we want. To misquote Churchhill, it's the worst form of communication except for all those other forms that have been tried from time to time.

[personal profile] betyoucanbetyoucant 2016-04-16 03:28 am (UTC)(link)
So far I've seen that AOL/AIM is letting you get alerts. That's the email I connected this journal specifically to after trying to get with gmail and hotmail.
supraliminally: (neutral -- blank look)

[personal profile] supraliminally 2016-04-16 04:42 am (UTC)(link)
Is there anything we can do on our ends? Like some kind of account setting to manually whitelist anything from your domain?
supraliminally: (calm -- eyes closed)

[personal profile] supraliminally 2016-04-16 05:44 am (UTC)(link)
Ah, that's too bad.

Well, thanks for keeping us updated on the situation. We know you're working really hard and we appreciate everything. Good luck.
bens_dad: (Default)

[personal profile] bens_dad 2016-04-16 06:46 am (UTC)(link)
Is there anything we can do on our ends? Like some kind of account setting to manually whitelist anything from your domain?

I don't use GMail, so don't know what you should tweak, but my understanding (I'm on a mail-manager's list where the GMail representative often explains how to sort this sort of thing out, as well as how not to get here in the first place) is that this *is* the best way to get gmail to like DW emails again.

If email from DW gets into your spam folder do tell Gmail that it got it wrong; the GMail artificial intelligence will learn that DW mail *is* wanted, then as more people say that they want DW mail their reputation will be restored.
marahmarie: my initials (MM) (Default)

Oh, wow

[personal profile] marahmarie 2016-04-16 05:53 am (UTC)(link)
People getting mail at Gmail are reporting email delays of about an hour.

For reasons related to Microsoft flagging DW's email as spam, I forward my DW email from Gmail to Outlook, which means the same delays, I guess, and that I have no provider left to switch to that I could tolerate (in fact, I can't tolerate GMail nor Yahoo, which is why I forward email to Outlook - the latter's interfaces are too garish, cluttered and arcane for me to deal with).

But I'm wondering how Dreamwidth is not whitelisted with all major email providers by now (it's been online for eight years! I think!). It seems weird that major email providers don't make more of an effort to ensure legitimate organizations such as Dreamwidth don't get flagged as spam senders.

And it's frustrating. If the bottlenecks continue with big providers as reported in the OP I guess I'll have to find a smaller provider to switch to (or buy a website to set up an email server like I've talked about but have never actually done) as Outlook is my favorite but has not, as far as I know, let DW's email through in the last year (edit: just checked my own DW and according to a post I made about it, it's been a year and four months since that particular problem started).
Edited (clarity) 2016-04-16 05:57 (UTC)
azurelunatic: A glittery black pin badge with a blue holographic star in the middle. (Default)

Re: Oh, wow

[personal profile] azurelunatic 2016-04-16 07:58 am (UTC)(link)
My buddy and I were just chatting tonight about spam blacklists and how hard it is to navigate even when you know what you're doing. He manages his own email server and subscribes to one of the less extortionate blacklists (there are many spam blacklists out there, and some are more reliable than others) and this week he got a call from his insurance agent asking whether he had the information ready to send over that the agent had asked for in his last email. Turns out that while the agency's automatic mass email stuff was getting through just fine, his agent was sending from home or some such thing, via an outgoing server shared with a lot of other customers ... and when one of the 1000+ other people using that server gets compromised and starts sending out spam, then everything sent via that outgoing server is suspect. And it had got onto the blacklist, badly enough that my buddy's mail server wasn't just dropping it in the spam bucket, it was just kinda dropping it without acknowledgment. He would have never known if the agent hadn't called, because other email from the same domain (but being sent via a different mail server) was getting through. Best of luck if you do wind up managing your own server! I salute everyone who makes the attempt, because the world needs people willing to make a go of it to see if they can.

Longevity doesn't necessarily mean that much when there's been a sudden change in pattern, since there's always the chance that a spammer compromised a legitimate provider and is now attempting to fire the spam-cannon double-quick before they get shut down and have to compromise someone else's mail server. I suspect that the bigger the provider, the less they care about individual email getting through as long as they can lower their spam volume and reduce their hours of labor to email traffic ratio. :\
marahmarie: my initials (MM) (Default)

Re: Oh, wow

[personal profile] marahmarie 2016-04-17 04:24 am (UTC)(link)
So basically I'd have to buy my own rack space (not just my own space on, say, a shared web server) to ensure I don't pay for the misdeeds done by others on my rack. I should have known there was a catch!

It might be a bit - controversial - to say so but the second I read the OP I wished email providers had not gotten so "good" at flagging and filtering spam (I use the word "good" loosely because while I'm sure that compared to maybe 10 years they catch some bigger, more obvious miscreants now that otherwise would have basically flooded out email servers and inundated customers with spam) because they suck at it, the proof being moments exactly like what Dreamwidth is going through now with GMail.

I think the good get punished with the bad and that that level of strictness is unnecessary when that's what your email account's Spam folder is for, and every email account everywhere has one, so why not just concentrate their endeavors on getting better at filtering into it instead of basically knocking the knees out from under so many other legitimate website's functionality?

Your buddy's story kind of highlights what I mean - why not reroute newly suspected spam to the Spam folder instead of into The Big Black Void? Who does that help? If indeed the email server was taken over nefariously wouldn't it be better (for your buddy's email provider, in the long run, if not necessarily for other customers in the short run) if your buddy could see something was amiss by the strange emails getting sent from that address and be able to report the problem himself? An argument could be made that people don't check their Spam folders often or carefully enough to catch such things but I think the success rate that way would still be higher than zero, which is the success rate they'll have picking the Big Black Void over anyone's inbox.
Edited 2016-04-17 04:26 (UTC)
marahmarie: my initials (MM) (Default)

Re: Oh, wow

[personal profile] marahmarie 2016-04-17 08:17 am (UTC)(link)
Yes, I agree...the same way you feel email is a broken protocol I feel (even more strongly that) HTTP is a broken protocol, and I can get really livid about it, so I hear you, believe me.

And yes, I took it as a a given that anyone might respond that the cost to users and email providers in terms of damage done by opening and clicking infected links in spam emails outweighs any benefit gained by not taking such drastic measures against it. Filtering to the Spam folder could be a great way to solve a few problems around this if the public was more educated and aware about how important it is to be careful but I guess as a practical matter that's never going to happen.

But I feel simple communication is not being utilized effectively enough to prevent a lot of the problems websites like your own can wind up having. I keep running the "friend at Google" scenario through my mind, in which you update DW's MX records, then notice an email slowdown and trace the cause of the delay back to the MX records update, but then you just contact a friend or person serving in some helpful official capacity at Google and bam! the problem just goes away. As it should!

Maybe this is not how the world works but it seems major email providers could have a list of safe senders with websites like your own included and when there's a problem like the MX records update, before they let that slow everything down there's a protocol in place where either they know to contact you or another known site admin to make sure everything is OK before slowing DW's email down or where you know you can get in touch with someone on their side to the same effect.

It's the turnkey-ness of the Internet that gets to me; you can look up someone's profile in seconds online even if you don't know them but major websites and email providers can't communicate with each other more effectively and I just don't understand why.

I mean, I'd bet there are people at Google right now who know what Dreamwidth's about, who can safely assume DW could not possibly be a spam sender unless it got hacked, who know Dreamwidth's like LiveJournal, who know sites like LJ are generally considered safe senders, so there is a shortcut way out of this through better communication between email providers and DW itself, it's just the providers (apparently) don't currently allow that communication to happen, relying on automation instead to (incorrectly) determine and (fail to) solve problems. Which is slowing up things on Dreamwidth for all of us and seems unnecessary.

As to email being broken, while I'm not quite as passionate about why that's bad as you are, I think it's bad enough that it should all be replaced with something like Slack. :)
Edited (typos, clarity) 2016-04-17 08:31 (UTC)
azurelunatic: A glittery black pin badge with a blue holographic star in the middle. (Default)

a friend at Google

[personal profile] azurelunatic 2016-04-18 03:19 am (UTC)(link)

It turns out that it's really hard to find out how much email traffic goes through gmail every day; at least, I wasn't able to find out directly.

According to http://www.radicati.com/wp/wp-content/uploads/2015/02/Email-Statistics-Report-2015-2019-Executive-Summary.pdf (stats for 2015, and predictions through 2019), there are over 205 billion emails exchanged per day.

According to https://emailclientmarketshare.com/ , gmail as a client has a market share of 16%, which doesn't cover the number of people using a desktop app or mobile device to access their email.

So we can guess that gmail is handling around 32, 33 billion emails per day.

I don't know what Dreamwidth's email volume is actually like, but http://www.dreamwidth.org/stats says that currently 4063 accounts have posted at least one entry in the last 24 hours. Let's posit that each of these journals has posted an average of 2 entries. I don't know what average comment stats are like, but I currently have roughly 30,000 entries in my journal, and 58,000 comments on those entries. I'd say that my comments are moderately active -- there are journals with less chatty readers, and roleplaying and socialization communities with far far more. But let's say that 2 comments per entry is average. Since I'm a paid user, I have signed up for a notification of every comment I send (as well as the notification that most people get when someone replies), and it's possible for people to subscribe to threads they're not a part of, so let's say that every comment left sends about 4 notifications. (Again, I don't know the real stats.)

So let's say: 9000 entries (rounding up), each generating 8 notifications. That's 72,000.

72 thousand is 0.00000218181 of 33 billion.

So at a guess, Dreamwidth is not such a significant blip on gmail's radar that a human being would ever be prompted to examine Dreamwidth's trustworthiness status, much less spend the time to select a form letter about changes to MX records and send it off to the address listed as the domain's administrative contact and review the answer when it comes back. (Assuming California minimum wage, that's about $0.50 for a 3-minute process.)

From my experience living and working in Silicon Valley, the "I know a person at $COMPANY" thing is calling in a moderate to major favor. This is what you do if you've already tried the official processes and there's genuinely no way forward, or if the official route leaves you dead in the water for a week or two. I don't have access to Dreamwidth's email stats, but from some of the patterns of grumbling, Gmail pitches some sort of woebucket over Dreamwidth's notifications maybe 4+ times a year.

Dreamwidth staff and volunteers do not, as a rule, help Google employees move house quite that often.
marahmarie: my initials (MM) (Default)

Re: a friend at Google

[personal profile] marahmarie 2016-04-19 04:27 am (UTC)(link)
Well, that does make DW seem like a tiny speck on Google's horizon. Thanks for digging up so many stats and doing so much math on that; those are some truly eye-opening results.

*is sad for our relative speckiness*

Where our thoughts kind of diverge is where you write, "So at a guess, Dreamwidth is not such a significant blip on gmail's radar that a human being would ever be prompted to examine Dreamwidth's trustworthiness status", because the way I see a possible answer to this is in the fact that it's not how big or small Dreamwidth is, it's how much can we trust the reputation, longevity, and current visible/knowable behavior of any website, regardless of its size.

If....if there was something like a cross between Google's rankings for websites (which more and more do seem to sift for trust, reputation and reliability factors, overall) and Web of Trust, but for email servers...I'm just not sure exactly how it would be done as far as nuts and bolts go but I'm thinking that unlike WOT it could be mostly automated/algorithm driven and then checked over by a WOT-like team, only perhaps a paid team with volunteers as opposed to the entire (unpaid) Internet that WOT draws from now to rate All The Websites.

And from this vague and fuzzy idea I'm having that's sort of hurting my brain to flesh out, a whitelist that stays updated in real time can be created, perhaps distributed to major email providers, and any checking against it for safe or unsafe senders, say when some website's MX records suddenly change, can also be automated, requiring followup only when a) the website is having other problems or b) other obvious red flags become apparent.

As to the friend at Google, it pains me to show my age by recalling the web as a much smaller place than it is. I had (to this day a completely unknown) friend at Wordpress who kind of helped me out back in the day (without me asking!) when even Wordpress wasn't too big (but it was getting pretty big!). Of course as a user of the site, blah blah blah...(I guess he or she was a a fan, but as they chose to remain anonymous, I'll probably never know exactly what inspired them).

So, there might be billions of websites of every stripe and flavor but there are only so many social media sites with longevity, good reputation and loyal users; even if Dreamwidth ranks at the low end of userbase among them, it surely must rank (actually, I'd be curious to know where on the scale of most-used social media sites it stands, but my gut feeling is it's at least pretty well known). Silicon Valley et al pays attention to social media and knows who's who, so it just rankles me a bit that communication can't be better than it is.
Edited (typo) 2016-04-19 04:31 (UTC)
azurelunatic: A castle with rockets and fire cannons with the DW D on it. (Castle Dreamwidth)

Re: a friend at Google

[personal profile] azurelunatic 2016-04-19 04:40 am (UTC)(link)
I have the opportunity to say where my Open Source involvement is, at work and at tech meetup events. When I say "Dreamwidth" I mostly get blank looks; maybe 1 time out of 100 or 200 I get a delighted "ooo Dreamwidth!" or a "Oh huh, are they still around?"; when I say "a code fork of LiveJournal" I get variations on "Man, I used to have a LiveJournal..." about 95% of the time. Sometimes they recognize Dreamwidth if I mention that it's one of the few open source projects which are majority women. So off the cuff I would not say that it is well known outside of certain fannish circles, which do not line up with Silly Valley dev circles in the way they may have some time ago.
marahmarie: my initials (MM) (Default)

Re: a friend at Google

[personal profile] marahmarie 2016-04-19 04:58 am (UTC)(link)
or a "Oh huh, are they still around?"

That's actually what I'm often tempted to say about, you know, the other half of that equation. Another of my gut feelings is LJ has been picking up steam lately and good for them, as I feel (with the cross-posting abilities we have in place that I see a lot of fairly popular DW users taking advantage of each day) that it might actually help Dreamwidth to grow, in the long run.

I can't really speak to how SV's great teaming masses react when DW's name gets dropped...when I say my gut feeling is DW is at least well known, I'm referring more to the upper echelon of Valleyites, whose job it is to kind of stay well-informed (at least I would think so) not so much their dev circles (though I have mad respect for some of them and think they might have at least some knowledge of us, too).
Edited (typos, incomplete sentence) 2016-04-19 05:22 (UTC)
azurelunatic: A glittery black pin badge with a blue holographic star in the middle. (Default)

Re: Oh, wow

[personal profile] azurelunatic 2016-04-17 08:27 am (UTC)(link)
At the least a dedicated IP; I'm a little fuzzy on how easy it would be to tell that two things were on the same physical box as opposed to just the same data center if they have separate IPs; that's a level of forensics a little beyond my expertise.

Even though my buddy's running his own mail server and gets to make his own rules, the straight-up blackholing of suspected spammers is helpful because it's basically a "woops, there's no mail server there anymore" message, rather than a "there's a mail server here but it says Talk to the Hand b/c the inbox ain't listening" (an explicit hard bounce message) or a "Heyyyyyyy we got your email bro!" (accept and send to spambox). Spammers who get a "heyyyyyyyyyyy we got your email bro!" will increase the amount of spam they spray at that server, since it's just indicated it's a possible target. So he's made that choice deliberately and from an informed place. Unfortunately this time it did in fact catch legit mail. So he sent the relevant information along to the agent so the agent could give it to his IT guy to go talk to the blacklist maintainers and get un-blacklisted.
Edited 2016-04-17 08:30 (UTC)
marahmarie: my initials (MM) (Default)

Re: Oh, wow

[personal profile] marahmarie 2016-04-18 01:27 am (UTC)(link)
I failed to take into account that letting spam through encourages spammers to simply send more spam. Thanks for explaining that as it illuminates a lot (admittedly, my ideas on how spam could be handled are likely naive and uninformed, which is why I ask a lot of questions and wonder openly about a lot of things - there are probably good reasons my ideas are impractical or just plain undoable but I can't envision what they are without looking a lot more into the topic).
azurelunatic: A glittery black pin badge with a blue holographic star in the middle. (Default)

Re: Oh, wow

[personal profile] azurelunatic 2016-04-18 02:29 am (UTC)(link)
Yup. Encouraging spammers: feeding seagulls is a better life choice!
marahmarie: Cockroaches get cancer, too (cigarettes)

Re: Oh, wow

[personal profile] marahmarie 2016-04-19 03:55 am (UTC)(link)
Never, ever feed the seagulls! If you practically grew up on the beach like I did then you know this is blasphemy! (Spent major chunks of my childhood/life on one or another.)

Barely any worse than feeding spammers, though, I must admit.
jordannamorgan: An axe-wielding skeleton in a baseball cap. (Mayhem)

Re: Oh, wow

[personal profile] jordannamorgan 2016-04-18 05:12 am (UTC)(link)
I just had to say that the phrasing of your examples was hysterical. :D Still making me giggle.
azurelunatic: A glittery black pin badge with a blue holographic star in the middle. (Default)

Re: Oh, wow

[personal profile] azurelunatic 2016-04-18 05:54 am (UTC)(link)
My excessively helpful brain has now started to ponder life as it would be if email servers had names and personalities like Culture ships.
putyoudown: (Default)

Re: Oh, wow

[personal profile] putyoudown 2016-04-16 08:17 am (UTC)(link)
I think Denise said something about this in the OP -- something about their most recent code push has basically made them look like a new email sender, so they were good and whitelisted before but now they're starting from scratch. It'll probably not take long to get whitelisted again, though.
marahmarie: my initials (MM) (Default)

Re: Oh, wow

[personal profile] marahmarie 2016-04-17 05:08 am (UTC)(link)
We changed some of our MX records

I was curious what that looks like (I last set up MX records 10 years ago on my own website, so my memory is long gone on that) so I googled some and found a checking service which says DW has only one MX record - http://www.intodns.com/dreamwidth.org. I'm not sure how to interpret the data as the website checking DW's information could of course be wrong, or DW could in fact have just one MX record, like it says. I'm passing along what it says just in case it in any way relates to the current problem, which of course I doubt, but just to be on the safe side...

And thanks for the warning on running one's own mailserver...if it's that much trouble I probably won't do it (and this might make people who know me laugh, but I was eying the comment up above about AOL not experiencing delays with a lot of hope...because as much as I don't like AOL, as a company, I think I *would* switch to their interface before I'd switch to using Yahoo's or Gmail's).

Azz was telling me a story above that kind of reflects what DW's going through now with GMail, only worse (mail was not slow - nope, it simply disappeared thanks to someone else's bad behavior on the same server!) which makes me realize I'd probably have to buy my own rack space just to avoid most of the possible issues. With running your own mailserver being decidedly more difficult than running your own web server, I see no reason to put myself through that (after all, dedicated rackspace is very expensive).

I hope all the email slowness gets resolved soon!
Edited (typo) 2016-04-17 05:29 (UTC)
blueraccoon: (Default)

Re: Oh, wow

[personal profile] blueraccoon 2016-04-19 08:56 pm (UTC)(link)
*raises hand* So I work in IT Operations for a Large Tech Company and I used to work in the group that helped manage our email, and there is just so much of it that it's really hard. Whitelisting only works for specific IP addresses or addresses; if those keep changing, or if the protocol used for them changes, the whitelist is no longer accurate and you have to update it. This is actually an issue we've had at work a few times, where the ops guys think something is whitelisted and it actually isn't and then things break.

Most email providers on the scale of Outlook or Gmail or whoever do have a process for getting yourself unblocked, but it's generally really opaque and there's no guarantee the results will last (as DW has seen).

Like you, I forward my gmail notifications to my Outlook account, so this just adds more delay but I'm out of providers and have no wish to set up a new account...
marahmarie: my initials (MM) (Default)

Re: Oh, wow

[personal profile] marahmarie 2016-04-20 03:37 am (UTC)(link)
Yeah. The email delays we're seeing are not fun, though. I checked how long last night as Azz and I were bouncing back and forth on this page and the delays were so long I was only able to see their replies when a) I couldn't edit my comment about whitelisting (as it had already been replied to) or b) I refreshed the page. So at the least 2-3 hours (I gave up trying to see how long it took to hit my Inbox after that).

On the face of it, email is pretty much a whitelist by default, as Denise says, so right now we deal with the problems that so easily causes by blacklisting (or otherwise kind of torturing) suspect senders. Spammers expect to be blacklisted so probably know enough to realize they need space at multiple servers/locations. I'm sure the more proficient among them just plan accordingly.

So blacklisting is not an incentive to not spam; it's an incentive to own more IP blocks/domain names/servers. Unless spammers run their own servers or use botnets (which I'm sure they do in addition to merely spamming from, say, their own publicly available websites) the only people incentivized to support blacklists are the ones selling spammers server space and domain names. Because spammers need a lot of both.

I'm thinking since email is basically whitelist by default and since blacklists can often be - I hate them, quite frankly I think they're bullshit - inaccurate, to say the least, not to say far from complete by the very nature of how spammers work tirelessly to circumvent them, that working more with the whitelist idea might be better in the long run.

Incentivize that; make it so everyone who sends email realizes they get priority treatment if they behave, but simply get vaporized (in the long run) if they don't (but if they do behave badly, then as soon as they correct their behavior send their email sailing right through again; nothing is permanent in this system because it simply reacts to how you behave, and does so as close to real time as technology will allow). No more blacklists outside of a record of which senders you vaporize. Automated algorithms pick apart the good from the bad and make the system impossible to game by flagging for enough signals to ensure the sender is legit.

If a legit sender such as Dreamwidth changes, say, a MX record and that throws a red flag then until it can be dealt with (again, perhaps in combination with the currently non-existent customer service we have so far) then rather than initially slow the email down through forced resends or outright disappearing it right off the bat, the email is linked to the same way a webpage can be.

So in your Inbox? You click a link to open the email and all you get is another link; when you hover that you basically have to sign a clickwrap agreement saying you realize the mail in question might destroy your computer, your life and steal you or your neighbor's firstborn and everyone's dog before you can even download it. Even then, images and links within the suspect email stay blocked with more warnings you have to agree to before you can open them.

This might solve some of the server space problems with storing spam Denise mentioned, as nothing suspect is shown to the user except the email title - not without extensive warning to the user on every suspect email and a very explicit agreement being clicked upon that the user does in fact understand things could go really, really bad. If the user doesn't try to open such emails within, say, two weeks, they're flushed from the server because reaching people's eyeballs through their inboxes is a privilege, not a right.

Besides that, just incentivize good behavior to make people want to remain on a whitelist and maybe force at least some of the less determined spammers to clean up their acts, so say, in return for good behavior, prioritize all their email delivery; over time as trust is built give them little Safe Sender icons like you see in Outlook for - I think it's eBay and Microsoft's own mail - to show the user whom to trust; offer reasonable paid subscription services that provide instant access to email tech support to - why aren't things like this done now?

I'm not sure why email seems to lack salespeople with some positive spin on things but it needs a way to unite the backend work of delivering safe email with the front end work of encouraging people to want to do exactly that. I can't be the first or the 10 millionth person to think so and most of my ideas are probably not original, it's just a remix of some I've seen and had over time.

Again, this is all a bit pie in the sky on my part and there are probably many reasons why what I'm suggesting can't work as one cohesive, completely secure program, but that's where I realize more minds on the same problem have either already discovered it can't be done or maybe just haven't iterated or reiterated it enough yet.
Edited (more info) 2016-04-20 03:44 (UTC)
jeeps: (btvs ♡ put 'em up)

[personal profile] jeeps 2016-04-16 05:58 am (UTC)(link)
I live about 10 minutes from Google, want I should go bust down their doors?
azurelunatic: California poppies, with a bright blue sky and the sun. (California girl)

[personal profile] azurelunatic 2016-04-16 06:22 am (UTC)(link)
Ooo, a field trip?! :D :D :D
saturnofthemoon: (Gaila)

[personal profile] saturnofthemoon 2016-04-16 06:05 am (UTC)(link)
Thanks for keeping us updated.

[personal profile] jazzyjj 2016-04-16 03:27 pm (UTC)(link)
Thanks for your excellent work as always. I've actually been wondering something for some time now. What exactly is a "subdomain" of gmail? Not to stray too off topic here, but that is the address with which I signed up on Dw in the end, and it also happens to be my work account. Lol don't tell them I said that. Actually they've been pretty cool with me working from home, at least ever since I started doing so. I'm using a web interface, but it's not at all cluttered like some of them are. At least not in the HTML view. I'm pretty sure that's how I'm accessing it. But back to Dw. I've not experienced this yet but if and when I do I'll report back. Btw, not a big deal about the subdomain question, but if one of you kind souls can offer up an explanation of this that would be cool.
azurelunatic: A glittery black pin badge with a blue holographic star in the middle. (Default)

[personal profile] azurelunatic 2016-04-16 06:48 pm (UTC)(link)
I'm not sure exactly what it means in context of gmail, but in the broader sense of domains, it's something like:

Say you own example.com. You want to have a main website, and you decide to put it at www.example.com. www is the subdomain that's set up for the website.

Say you want to have a different website on a completely different server. The easiest way is to set up a separate subdomain, something like website2.example.com. website2 would be the subdomain.

Commercial email service providers

[personal profile] babysprite 2016-04-16 05:48 pm (UTC)(link)
Each seems to "do its own thing" and maintain it's own set of inter-networking standards.

I stay with my ISP email, read and send correspondence in plain text.
dollie: (Default)

[personal profile] dollie 2016-04-17 03:18 am (UTC)(link)
I guess it's not just Gmail. I use Yahoo for all my accounts and I get notifs at least 6-8 hours later, depending. Hopefully things work out and good luck.
talkswithwind: (tech)

[personal profile] talkswithwind 2016-04-18 01:34 pm (UTC)(link)
My deepest sympathies on email deliverability problems. That kind of thing falls into my Dayjob responsibilities, so I know those feels.

I noticed that dreamwidth.org doesn't have a DMARC entry yet. This is Yet Another Thing to say to mailers, "no, really, I'm me". It works along with SPF and DKIM, which are already set up.

Last month, Amazon SES added support to customize the MAIL FROM: address in the SMTP conversation. This is a big thing for SES customers, as it makes DMARC much easier to set up. Create an SPF record in the 'dreamwidth.org' domain, set it up right with -all at the end, and deliverability would increase a bit. Better yet for troubleshooting, gmail will put a helpful additional header in messages saying how well incoming email passes DMARC, SPF, and DKIM

dmarc=pass (p=NONE dis=NONE) header.from=dreamwidth.org


Pretty handy.

One caveat I know from bitter experience. Setting a policy to enforcing will break mail-forwards. People who have gmail addresses but forward email off of gmail will generate bounces like, "Mail administrators at $domain have administratively prevented this email from delivering," which will break things. Great for platforms where nonrepudiation is more important than deliverabiity, not so great for platforms where deliverability is paramount.

azurelunatic: A glittery black pin badge with a blue holographic star in the middle. (Default)

[personal profile] azurelunatic 2016-04-18 08:54 pm (UTC)(link)
One of the things that has turned out to create difficulties is that Dreamwidth offers an @dreamwidth.org forwarding address as well as sending notifications from an @dreamwidth.org address. Therefore a lot of the stuff which could be done to verify Dreamwidth's notifications would screw up that.

`\_(ツ)_/`
talkswithwind: (tech)

[personal profile] talkswithwind 2016-04-19 03:10 am (UTC)(link)
Putting you in the awkward position of being both an automated mail-generator, as well as remailer. That's a tough position, and there is no way to keep everyone happy. We can't use enforcing-mode DMARC at DayJob because we need deliverability far more than anti-phishing; all it really does for us is tell other mailers that subdomains should be bit-bucketed.

Waiting for IP reputation to regenerate sucks a lot.

[personal profile] sockiness 2016-04-19 05:28 am (UTC)(link)
The incredibly painful thing about all of this is that there's windows of normal delivery and you think 'hey, maybe it's gotten back to normal', then you have an hour to six/seven hours up of delay, and it sucks.
burn_the_world_down: (Drinking)

[personal profile] burn_the_world_down 2016-04-19 06:36 am (UTC)(link)
Have we heard any word about this? I know you are trying, but it is starting to get a little old, getting everything on time, then going back to not getting anything. It's not even consistent anymore, the delay.
burn_the_world_down: (BW Grin)

[personal profile] burn_the_world_down 2016-04-19 06:03 pm (UTC)(link)
Thank you! I will actually give that a try and see if it clears the inconsistency.
zunge: (Default)

[personal profile] zunge 2016-04-19 03:19 pm (UTC)(link)
I am noticing this problem with yahoo mail. Notifs are late by anywhere from fifteen minutes to several hours.
blueraccoon: (Default)

[personal profile] blueraccoon 2016-04-19 08:45 pm (UTC)(link)
Did the issue with Hotmail/Outlook ever get resolved? I switched to Gmail for notifications when Outlook listed DW as a spammer, and this post just made me think about it.
blueraccoon: (Default)

[personal profile] blueraccoon 2016-04-20 07:19 pm (UTC)(link)
Then I will test it and report back. :)
blueraccoon: (Default)

[personal profile] blueraccoon 2016-05-02 10:14 pm (UTC)(link)
I am reporting back to let you know it did! I switched my notifications back to my outlook account and although it took a few hours to get the validation email, I'm getting notifications without a problem now.

[personal profile] sockiness 2016-04-20 01:15 pm (UTC)(link)
I've been through a lot of code pushes here and I'm generally used to notifs being borked for the day of/into the next day after something pretty big. This, however, takes the damn cake. And no, I'm not blaming you at all, but I'm sort of beyond frazzled now that this has gone on as long as it has (as I'm still getting windows of both delays from an hour up to 3+ as well as blips of normal notif time) and no real way to contact gmail and tell them to smarten up beyond leaving feedback (which I have) and just sitting here waiting. If this is them being 'unsure' about a site, they can just... I don't know. Figure out a way to validate quicker, because anything over 3 days is bullhooey and this is well past that. (again, not blaming you, but just venting a wee bit)
nopesock: nope (Default)

[personal profile] nopesock 2016-04-27 07:03 am (UTC)(link)
Okay, you know what? This is obviously not going to get fixed. It's time to shitcan support for gmail on this site.

Page 1 of 2

<< [1] [2] >>