Email issues to Hotmail continue, joined by issues to Yahoo

[denise]

For the past week and a half or so, Hotmail has been rejecting all email from us. Sadly, Yahoo has now joined them. This means that if your confirmed email address is @hotmail.com or @yahoo.com, you won't get any email from Dreamwidth at all. (This includes comment notification emails, password reset emails, lost username emails, confirmations/receipts for payments, email from your @dreamwidth.org forwarding address ... basically, if it's email and it's from us, it won't reach you.)

This is a block at the system level, so there isn't anything you can do to work around it. It's not due to spam filters on your individual account, so whitelisting Dreamwidth in your mail application won't work to fix things this time: these providers are blocking all mail from us, across the board. When a provider makes that kind of block, they don't discuss the exact reasons that led to the blocking, but our best guess is that it's due to a combination of the amount of email we send out (especially email with highly similar subject lines and content, since -- for instance -- a notification of a new comment to an entry quotes the entry) and an uptick in the amount of spam sent to @dreamwidth.org forwarding email addresses, since that forwarded spam can look like we were the ones to send it.

We've gone through the process to request unblocking from the providers directly, but haven't seen any progress there yet. We are, however, actively working on alternate solutions that will reduce the risk of us being identified as spammers in the future -- we ran the first test yesterday and results are looking promising. We'll keep you posted on the progress.

Comments

[runningred]

Thank you so much for being on top of this! Your hard work is much appreciated.

[sharpiefan]

Ugh.

I rely on my notifications coming through, and though I've not noticed any issues, as a yahoo.co.uk user, I really hope this gets sorted before UK Yahoo joins Yahoo.com in blocking stuff.

Thanks for the warning and for the work you do in keeping DW running as smoothly as it does. We all really appreciate it!!

[yuuago]

That really sucks. Thanks for the information; hopefully there'll be progress on getting it unblocked.

[deathscore]

Have you considered preemptively shooting a note to gmail about this? It's the provider most people I know use anyway, and if we have at least one working people can at least transfer their accounts there.

[goodbyebird]

Sucks that email providers are so hard to get a hold of. Thank you for your hard work though <3

[bluerosedreams]

Do you know if this has gone through yet or not, or if it's a future thing? I just tested this by sending an invite on one of my counts to join my musebox, and the notif went through fine.

[nuranar]

My ATT email has blocked DW for 2+ years. Is that something you'll consider pursuing?

+1

[lampdevil]

I recently switched all my stuff to Yahoo because of the Hotmail issue, and I still seem to be getting notifications there just fine. Maybe they're still rolling the block out to all their servers, or something to that effect....?

[bluerosedreams]

My first thought was that they (stupidly) have an automated filter and quickly caught the mistake and rectified it.

[annejack]

hmmm, I do get comment notification, just not your updates (at least this one I haven't gotten in my e-mail)

[denise]

We've been through gmail's process before (when we first started up), and they're generally pretty good at not blackholing you twice unless you wind up on one of the industry-wide blocklists.

[denise]

Nope, we're definitely seeing blocks from Yahoo. It's a soft block and not a hard block, which means that mail is delayed but may make it through on the second try, though.

[mdlbear]

Maybe a form of greylisting, then?

[pineapplesalmon]

I have heard that yahoo accounts seem to be letting DW through now.... Or only one yahoo exchange was blocking? I've heard some users never seeing a problem, even state side.

However, I've been told that Hotmail routes through the same exchange as Live and Outlook, so they are all blocking notifications, most likely. At least one Live user had to switch to gmail to get their notifications again.

Hope this info helps!

EDIT: And then I read your comment about soft blocking from yahoo. That explains the discrepancies!

[denise]

Sadly, we tried with AT&T (and their subsidiaries that use the same rules -- I think it's AT&T, BellSouth, and SBC Global) for a very long time and were unable to convince their systems that we were legit mail senders. (If I remember correctly, we managed to get unblocked after a few months, for a grand total of about two weeks before they blocked us again.) They're very aggressive in blocking mail senders, and very reluctant to unblock again.

The alternate system we're in the middle of implementing may work better for you, but I can't guarantee it. Sorry!

(Email is the absolute worst technical problem to solve. The worst.)

[ashkitty]

Earthlink does this too (even if I am possibly one of like five people in the world who still use it)--that's why I couldn't get much use out of DW until I started using a gmail account too. Don't know if this information is helpful, but in case it is, I leave it here. :)

[blueraccoon]

Hotmail, Live, Outlook, and MSN all use the same infrastructure, so if one blocks they all block. I have an outlook email address and had to switch DW mail to my gmail account to get notifications again.

Note to denise: I did some poking around on my side, but apparently I am powerless to change anything or put a bug in the ear of the people who are. >.< I'm sorry.

[moonvoice]

Not super relevant, but the massive online 'dragon collector' site Flight Rising has just been flagged massively and aggressively by Hotmail as well. So it definitely looks like it's an issue at least across more than one website. But I do hope there is progress that happens not just through your hard work, but through theirs as well.

[rosacaerulea]

Hm. :|a;

I've only got two out of ten notifs missing, and all of the ones I got came through immediately. So IDK. Hopefully it resolves swiftly.

[pineapplesalmon]

Do any of them have a twitter account? Maybe telling them over a public facing account might help.

(This is more of a general comment to everyone than to you directly, but thank you for the confirmation that they're all blocking!)

[sine_nomine]

Interesting; I was pondering switching my DW mail from Yahoo to my Mindspring account. Guess that's out.

Thanks for taking the time to post (from one of the two people who still use Mindspring even after Earthlink ate it)!

[denise]

Sadly, that really won't get anywhere. The handling of mail refusal and blocklists is done through entirely separate channels than the usual system for problem resolution, and front-line customer service not only has no way of addressing or resolving it, but often doesn't even know that the blocking is done in the first place. (They will invariably tell you to adjust your spam filters and/or your whitelist, when the problem isn't on an account level: it's at the system level.)

There are ways for a mail sender to request removal from a mail provider's blocklist. We've done that requesting, but it often takes time for those requests to be processed, and it's not automatic. There are ways for a mail sender to reduce the risk that their email volume will be treated as spam, and we do most of them, but they're not perfect.

Ultimately, the problem is that email is a protocol that dates from the days when every domain on the internet was run by somebody who knew most or all of the other people running all the other domains on the internet. If somebody was abusing the network from a particular domain, they could pick up the phone and call the administrator of that domain and say "hey, jane@example.com is sending spam, can you go down there and explain to her that we don't do that?" and the admin would. It's a protocol that, at heart, fundamentally assumes that every actor in the system is a good-faith actor. Security, authentication, and anti-abuse measures have been bolted on top of that basic protocol throughout the years, but upwards of 80% of all email sent is still spam anyway.

Most companies that run mailservers, especially the ones that provide free mail service to people, are very, very aggressive in their spam identification, flagging, and blocking, because they lose millions of dollars a year to the garbage. This does unfortunately result in legit actors like us getting swept up in the mitigation efforts sometimes, but there's a real battlefield mentality in a lot of the people doing that work, because spammers are incredibly inventive and very good at working around whatever countermeasures are put into place. Free webmail providers in particular are the most likely to be super-aggressive in blocking suspected bad actors, but everybody does it, to one extent or another.

Other websites don't have the problem as sharply as we have (although LiveJournal has the exact same problem, and for the exact same reasons, and has for fifteen years -- I remember handling this same issue on and off for nearly the entire time I was working there), but the fact that we offer the @dreamwidth.org forwarding email addresses makes mail providers more likely to block us: spam sent to those email addresses is forwarded just like everything sent to those email addresses is forwarded, and because of that, to mail providers it can look like we're the ones sending the spam.

We've talked over a lot of possible solutions for that problem (and, as the post says, are in the middle of implementing one of them; hold tight for more info on that) but ultimately the only way to permanently reduce the risk would be to remove that feature entirely, which we don't want to do because a lot of our users rely on that forwarding email address as a form of anonymity and identity protection. (And even removing that feature wouldn't remove the risk entirely, just reduce it: the very high volume of email we send, the fact that a lot of our email is nearly identical, and the network reputation factor that a lot of providers use in their spam heuristic -- that is to say, if there's a spam-sending machine on a particular hosting provider's network, some mail providers will block or downscore all mail sent by any machine hosted on the same network -- mean that this sort of problem will inevitably come up again.)

I know it's frustrating to be told there's nothing you, as a user of a particular service, can do to resolve an issue that's happening with that service! But in this case, there really isn't anything that anybody can do. Unless you or anyone reading this works in Hotmail's postmaster department, in which case, our request for whitelisting is in your queue and we'd really appreciate it if you looked at it. ;)

[madeupofstars]

I have extensive experience with IP level email deliverability, and may be able to help you with both your hotmail and yahoo issues (and cut any gmail issues off at the pass) - I'm not sure how to contact you directly (?) but I'd be happy to help / advise.

Yahoo mostly doesn't deal in on/off levers; it has to do with IP reputation (among other things, not going into technical details here). Please let me know if I can help further. (I do this professionally with a mailer who sends magnitudes more email than you do, guaranteed.) :)

Also just a note: 95% of the time, trying to contact hotmail, yahoo, or gmail directly will get you absolutely nowhere, in my experience. =/ They're just far too large; in order to have actual account reps, you're looking at needing to send hundreds of thousands (or more) email/day.

[denise]

Hopefully we are in the process of cutting the problem off at the pass, but thank you, your offer is super appreciated! OK for me (or, really, mark -- he's the one doing this) to email you at the email address you have confirmed on this account if he thinks picking your brain would be useful? (Or, you can email us at our forwarding addresses -- mark and/or denise at dreamwidth.org.)

[madeupofstars]

I'll pop an email to you both. :) Thanks!

Just sent it, it's from my professional email. I hope it works out well! :)