Denise (![[staff profile]](https://www.dreamwidth.org/img/silk/identity/user_staff.png)
denise) wrote in ![[site community profile]](https://www.dreamwidth.org/img/comm_staff.png)
dw_maintenance2023-09-28 11:16 pm
denise) wrote in dw_maintenance2023-09-28 11:16 pm
Continuing dispatches on the war against spam
A few days ago we let you know about spam prevention measures that we were taking to help stem some of the flood of garbage. One of those temporary measures included geoblocking all IPs from several of the countries that are our largest source of spam. This did (as we knew it inevitably would) have some collateral damage for real users, and we're very sorry!
We're continuing to experiment: this time we've slightly expanded the range of countries we're geoblocking to include the ones that we held off on geoblocking because it would affect too much legitimate use, but we've limited the geoblocking only to the account creation page. This should mean that if you were having trouble accessing the site because of geoblocks, you should be able to access 99% of the site without a problem, and the only page you won't be able to access is the account creation page. With luck, this should cut back heavily on our spam account creation without disrupting legitimate use of the site. The current list of countries that are geoblocked from account creation are Bangladesh, Cambodia, Egypt, India, Indonesia, Morocco, Pakistan, Singapore, Turkey, and Vietnam. (If you're an existing user from one of those countries and you'd like to make an additional account, email support@dreamwidth.org with the username you'd like to register and we can register it for you. If the number of requests gets to be enough that it's taking up too much of our time, we may have to pause this until we can build automated exceptions, but we'll start there.)
We will continue to monitor the results of these experiments and adjust as necessary: when we do one of these experiments, we always make sure to define in advance what "too much interference with legitimate use" will look like, and we try very hard to stick to it. I apologize to everyone who's been collateral damage in our efforts to filter out more of the goddamn spammers.
We're continuing to experiment: this time we've slightly expanded the range of countries we're geoblocking to include the ones that we held off on geoblocking because it would affect too much legitimate use, but we've limited the geoblocking only to the account creation page. This should mean that if you were having trouble accessing the site because of geoblocks, you should be able to access 99% of the site without a problem, and the only page you won't be able to access is the account creation page. With luck, this should cut back heavily on our spam account creation without disrupting legitimate use of the site. The current list of countries that are geoblocked from account creation are Bangladesh, Cambodia, Egypt, India, Indonesia, Morocco, Pakistan, Singapore, Turkey, and Vietnam. (If you're an existing user from one of those countries and you'd like to make an additional account, email support@dreamwidth.org with the username you'd like to register and we can register it for you. If the number of requests gets to be enough that it's taking up too much of our time, we may have to pause this until we can build automated exceptions, but we'll start there.)
We will continue to monitor the results of these experiments and adjust as necessary: when we do one of these experiments, we always make sure to define in advance what "too much interference with legitimate use" will look like, and we try very hard to stick to it. I apologize to everyone who's been collateral damage in our efforts to filter out more of the goddamn spammers.
no subject
2) Did you try to use mobile phone/SMS verification for Dreamwidth account registration?
3) Did you try to use Dreamwidth invites in order to connect new users to existing Dreamwidth users?
no subject
Also big privacy issue.
no subject
no subject
no subject
no subject
no subject
Yubikeys can also be used with an authentication app that can be used on non-phones, if you don't have a smartphone!
no subject
Discord
SMS verification
SMS verification does not have to be mandatory.
It should be OK to maintain an old Dreamwidth account without connection to any phone number.
Other alternatives could be invite-based registration and registration from an IP network that is not blacklisted.
no subject
no subject
no subject
I agree: an expensive SMS messaging in multiple countries - is a valid concern.
Then the filtering solution could be:
1) Invitations from existing Dreamwidth users.
2) Corporate email addresses (instead of freely available email addresses such as
3) Allow creating accounts to anyone, but severely limit the new accounts until they prove themselves and then blacklist IP address and corporate email domain in case if new account turned out to be spam.
no subject
2) In addition to being a privacy nightmare, SMS authentication costs a significant amount of money for a service to provide, is plagued by an extremely high rate of toll fraud and pumping (especially in the countries that we're having the highest spam problems from), and has a very low rate of success at stopping any kind of spam because any spam network of any appreciable size has racks of hundreds of thousands of SIM cards wired into devices and software that automatically tracks which numbers have been provided to which sites, selects the next number that's up, and handles the SMS verification. You can buy a full turnkey spam system for about $1000 USD. The only reason anywhere uses SMS verification anymore is inertia: it is useless.
3) If you read the comments on the previous post, you will see I have already answered this question.
Please assume that we are competent professionals who know a great deal more than you do about the problem and are addressing the issue in the least restrictive fashion possible to us that is likely to be effective.
no subject
All WiFi networks that are easy to hack into and are frequently used by spammers - you can automatically blacklist: every time you ban spam account - you should also ban the IP network that facilitated in this spam account creation.
> SMS authentication costs a significant amount of money for a service to provide
Sending SMS in the US starts from $0.0079
However I see that in other countries sending SMS is [10x?] more expensive.
So I guess SMS is not a good solution for, say, Vietnam, which IP networks Dreamwidth blacklists.
> toll fraud and pumping
The first article on SMS pumping fraud I found - describes the solution as well:
https://support.twilio.com/hc/en-us/articles/8360406023067-SMS-Traffic-Pumping-Fraud
But I see that Twilio's solution will not cover the most problematic countries such as Vietnam.
I guess that Dreamwidth solution for new accounts Vietnam, could be carefully monitored invitation system.
Another component of monitoring - reports from well-established Dreamwidth users.
If old Dreamwidth user friended the new account - it is a good sign.
If old Dreamwidth user banned the new account - it is a bad sign and Dreamwidth may automatically delete such new account. Especially if this new account was created from low quality IP network.
no subject
no subject
no subject
Let me try this one more time: the people running these networks are not using hacked WiFi or single internet connections or even IP addresses that have poor reputations in any available system that a site can hook into any kind of reputational check. We already do the things you are suggesting, and we have for years. These tactics no longer work, because spam is a trillion-dollar industry and there is more than enough money sloshing around in the ecosystem to build systems that evade even the most sophisticated reputational checks. The people who run these networks have a massive supply of IP addresses from multiple netblocks, from multiple providers, that are completely clean in every reputational database, and they use them until they start building up negative reputation and then pass them along to someone else to deal with the shit reputation while they move on to the next group of clean ones. I guarantee you that if I pulled the IPs of our last 100 spam accounts, every single one of them will have a cleaner reputation than the IP address you are currently using. Because they stop using them when they start accumulating negative reputation.
We are not talking "hacked wifi" or "scam ISP" here. We are talking people operating out of buildings that have multiple network drops from multiple providers with multiple completely clean reputations. For shits and giggles I tracked the network origins of our spam account creations from Bangladesh for about a week, because Bangladesh has a small number of ISPs, they're all licensed by the government, and you can get a list of the current license holders and therefore an accurate and complete list of all ISPs operating in the country. At the time I did that, there were about 130 ISPs licensed to operate in Bangladesh. We saw spam from over 100 of them.
Likewise, if your only understanding of toll fraud comes from a Google search and reading a surface-level article by a provider of SMS verification services with a vested interest in assuring the reader that the problem is not as bad as it actually is, you do not understand the problem well enough to be making suggestions for it. Up until Elon Musk bought it and laid them all off, Twitter had a team of about 20 engineers whose full-time duty it was to minimize the amount of money Twitter loses annually to toll fraud, which, as of my last information (since everyone I knew at Twitter got laid off) was eight figures a year -- and that's with aggressive engineering work to detect and prevent it. Add that to the fact SMS verification does nothing to actually prevent spam, because the networks we are talking about have access to an infinite number of phone numbers and the hardware setups needed to swap them instantly. SMS verification is not a spamfighting tool and hasn't been since at least 2015. It is expensive, it is a privacy nightmare, and it does nothing to fix the problem.
Site behavior is also not an accurate spam detection system. We already do it! We have for years! It detects less than 5% of spam account creation, and some days less than 1%. We have spent the last two years building increasingly sophisticated detection and prevention systems, both using in-house tools and demoing various externally available systems. We have found one option that was better than a 50% hit rate, and it was a) around 60% and b) also would be a massive privacy nightmare to actually implement.
You do not understand the scope of the problem, the sophistication of the operations that are involved, the cost of every single system that exists to address the problem (and how bad every single one of them are at actually detecting spam), or the sheer volume of garbage we're talking about. Between Mark's and my contacts, we probably know at least half of the top 100 people in the world at dealing with this problem at scale, and we have talked the issue out with them extensively. The suggestions you are making are easy, obvious, and don't work to solve the problem because they are the attack methods everyone was using to fight spam a decade ago and the spammers already adapted to them.
no subject
I am suggesting to use your own reputational database of IP addresses, and keep your own database private.
So it will be hard for spammers to find out if their IP address is already blacklisted.
Bad IP should not prevent Dreamwidth account creation, but instead should allow spammer to create the account, so Dreamwidth can collect other spam indicators, such as:
- Email address and email domain.
- Connections to other Dreamwidth accounts.
- Content keywords.
- Other involved IP addresses.
- ...
> move on to the next group of clean ones
Do you mean that it is easy for a scammer to get access to clean IP addresses?
The spammer's dilemma is that if IP address is easy to access - then this IP address is quickly getting blacklisted.
> if I pulled the IPs of our last 100 spam accounts, every single one of them will have a cleaner reputation than the IP address you are currently using
Does Dreamwidth maintain the internal database of IP addresses Spam/Ham scores (based on Dreamwidth users activity)?
> Because they stop using them when they start accumulating negative reputation.
If Dreamwidth does not immediately delete spam accounts, then it may be quite tricky for spammers to detect that their IP address accumulated negative reputation in internal Dreamwidth database.
> there were about 130 ISPs licensed to operate in Bangladesh. We saw spam from over 100 of them.
So penalize IP addresses from 100 Bangladesh ISPs and do not delete accounts created from the remaining 30 Bangladesh ISPs.
This will put users' pressure on the bad ISPs to deal with spammers in their own IP networks.
> if your only understanding of toll fraud comes from a Google search and reading a surface-level article
I run a job board and deal with spam and scam every day.
Spam is a relatively minor issue for us vs scam (which is operated manually and not on a bot scale).
For spam indicators we use:
1) IP addresses (and networks).
2) Email addresses.
3) Content keywords.
4) Browser User Agents.
5) User's feedback.
> Site behavior is also not an accurate spam detection system.
> It detects less than 5% of spam account creation, and some days less than 1%.
What do undetected spam accounts do?
If they do something harmful - why you cannot detect such harmful behavior?
no subject
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
no subject
no subject
The more tricky problem is how to treat legitimate IP addresses that are abused by spammers.
no subject
Nice catch, comrade major...
no subject
no subject