backend errors

[denise]

Someone reminded me we hadn't made a top-level post about the "backend fetch error" that people have been getting in the evenings -- this is a problem related to site load. (It's also related to the increase in abusive traffic we've been seeing over the last several months or so, and we're not the only ones who've been noticing it; there's been a general internet-wide upswing in garbage lately.)

We're doing our very best to keep legitimate site usage from noticing the problem. When you get any kind of site capacity related error, wait a few minutes and try again: our systems have a number of ways to automatically fix the problems, but it takes a few minutes for them to get more capacity online. Sometimes it can also take a few rounds of "add more capacity, see if that fixes the problem" for the issue to fully resolve. We've been looking into long-term fixes that we can make to reduce the amount of garbage traffic we're getting and the chance that it will overwhelm our capacity, but we're already getting feedback from legitimate site users that the traffic filtering steps we're taking are causing issues accessing the site for them, and we're trying to avoid making that problem worse.

We're really sorry about the hassle! Things may be a little rocky in the evenings US time for the next week or two as we work to find the best long-term solutions, because evening in the US is both a higher period of legitimate activity for us (because of our high number of US users) and a higher period of abusive traffic (because it's the beginning of the day in the countries we get the most abusive traffic from). If you do get an error, waiting 3-5 minutes and trying again should resolve it.

Comments

[dewline]

This news does explain a thing. Much appreciated!

[yuuago]

Thanks for the update! Glad to hear you're working on it.

[veritas_poet]

Thanks for keeping us updated. I thought something was wrong with my computer!

[bluedreaming]

Thank you for the update! This is very helpful to know.

[sixbeforelunch]

there's been a general internet-wide upswing in garbage lately

Is there a known reason why? My uneducated guess would be AI.

Thanks for the hard work! ♥

[denise]

The majority of garbage traffic ultimately leads back to cryptocurrency scams these days, with scammers scraping social media to look for accounts that are vulnerable to compromise (to use for scamming people) or to get profile data they can mix-and-match reuse to make their scan accounts look legit. (The second biggest garbage traffic category is SEO backlink generation spam, the bane of my freaking existence.)

[silverflight8]

Thanks for the extra background!

[tornir]

Does DW have/can it be equipped with swear-filter type capability?
Identified SEO backlink domain names get replaced with "SPAM" on posting, so it doesn't matter if the Google bots scan them before you can remove them; the spammer's already failed.

[denise]

We've avoided adding anything like that up until very recently because of how easy it is for it to over-trigger and interfere with actual posting, and we have a lot of people who post about technology and security who might have legitimate reasons to post specific domains and the like. We have been adding additional tools for dealing with it lately, just because of the extent of the problem, but about 80% of our spam is either one-off/small-volume backlink spam for new domains that haven't been established as bad already or a case of "15,000 auto-generated affiliate domains all forwarding to the same website".

There's also the problem that backlink spammers don't actually check if a particular website is giving them any real Google juice -- we actually had the Google crawler banned for a few months as splash damage from some of our bot detection systems before we could figure out a full list of the networks and user-agents they use for it, so no DW accounts were getting crawled at all for a while. The spam rate increased even though they were demonstrably getting zero benefit out of it. :/ Like 5% of the people who do this kind of thing are actually competent and clever and really good at working around restrictions you place on backlink spam, and the other 95% are warm bodies following a tutorial one of the 5% made and posted to one of the malicious SEO tactics forums. The 5% change up their tactics quickly, but the 95% never bother to check the dates on the tutorials. I've found some of the ones that people are following to spam us and we've changed up some things that make following those tutorials harder, but a lot of those kinds of tutorials live behind subscription services and I'm not giving money to malicious SEO forum operators just to figure out how they're telling people to abuse our site, heh.

[verushka70]

I'm wondering how much of this is related to exponentially increased use of AI... like, AI bots doing this kind of thing. I don't think it would have exploded as it has if humans were doing this manually.

I fear eventually we'll get to a point where much (maybe most?) Internet content is created by AIs and only consumed by AIs...

[denise]

Nah, backlink spam has existed for ages, and this wave we're in right now has been going steadily since before generative AI reached the market -- even today, the vast majority of the sites being spammed for are content that was written for pennies by actual humans working in countries with a favorable exchange rate, and almost all the spam that's posted to try to drive traffic to those sites is posted by actual humans working for pennies to create accounts and post it. We as an industry are reasonably decent at the arms race of bot detection and removal, in that we can detect and remove content that was posted through automation, but we're bad at identifying "garbage content that was posted by an actual human" because it's often indistinguishable from "not-garbage content that was posted by an actual human". Our spam detection on DW is multi-faceted and has a lot of different components, but "me and/or Jen looking at every account to determine if it's real or garbage" has such a higher success rate than any automated solution it's not even funny, and you wouldn't believe how often the spammers we suspend write in to us and ask why their account was suspended. The core of the problem is that it's easy to hire people for cheap to create garbage content farms that rank highly in search engine results, and advertisers will pay a lot of money for their ads to appear on pages that are highly ranked in search engine results. The difference between what an unethical content farm operator can generate in advertising revenue and what it costs them to pay actual humans to both write the content and spamvertise it (to drive up the search engine rank) is large enough that paying real humans in countries with a favorable exchange rate and a lower cost of living to generate the content and spamvertise it is still very profitable.

I don't blame the individual people who are doing the content generation and spamvertising, because it is a very good living for them, but even if you take them out of the equation and outsource that entirely to generative AI models, it's not going to make much of a difference in the volume, because the volume is already overwhelming. The problem is the advertising based model of revenue generation and the perverse incentives of the costs being shifted to platforms while the revenue goes to the people who are abusing the platforms, and I don't know what the solution is, honestly. We're small enough that manual detection and removal still works, but there's only so far we can extend that (especially since that manual detection and removal requires the person doing it to also have access to sensitive user data on the real accounts that are mixed in to the garbage ones, which means we're limited in how much we can let people help us with because we can't just give out the permissions to do it to anyone who wants to help). It's a giant fucking mess and the spam problem has been around for decades by now without a solution, because it's a really fucking hard problem.

[verushka70]

Ah. So it's not AI spambots, "just" spam resulting from late stage capitalism. Yay...? 🙃😑

[adore]

Thank you for updating us! I was confused, so this is good to know.

[bethctg]

Thank you!

[juan_gandhi]

Thank you so much for all that you do. We can be patient (at least I can).

[otter]

I did get that message today. And waiting a few minutes worked. Thank you for all the work you do to keep this place running. The people I interact with here are invaluable to my well-being.

[sheliak]

Good to know what's going on!

[cora]

Thank you, Denise!

[beach_baby]

Ahh i was wondering what happened!

[mistressofmuses]

Thank you for the update! And thank you for doing what you can to manage the abusive traffic problem.

[torachan]

Thanks for the update. I had been wondering.

[makamu]

THank you for the update and the additional info on abusive traffic.

[kore]

I was just about to ask about this, and here is this post! I always appreciate DW's complete transparency and detailed explanations of exactly what happens.

[vriddy]

Thank you for the explanation and background! Good to know.

[ysilme]

Thank you for letting us know! ♥

[sathari]

Thanks for the update and explanation! The first time I got the backend-type errors was right after I did a bunch of updates/other routine maintenance on my computer and I was afraid that was the problem, so this comes as a great relief. Seriously, I'm so grateful for, well, DW generally, and this kind of transparency, not to mention the reasons for it (namely, that you're trying to keep us end-users from having to deal with various sorts of nasties) is just one of the reasons why.

[smallrobot]

Thanks for the info and generally for this place existing.

[dhamphir]

I just want to say thank you, not only for what you're doing to address this issue, but everything you do to make Dreamwidth what it is! You guys are always on the ball, and good about being transparent and keeping us updated. That means so much! I've obviously not been to every site on the internet, but I don't believe there's any website that stacks up to Dreamwidth, and that's because of you guys. So thank you! :)

[denise]

Thank you! I am grateful to the user who poked me and reminded me that I hadn't made the post yet -- we try our very best to keep y'all up to date, but that raging case of ADHD means I don't always remember that I should post about a problem. So if you're ever wondering about a thing, please do reach out to us and prod me to explain it! Our goal for DW has always been "we'll keep it running so you don't have to worry about it, but if you want to know more about what's going on/our reasons for doing a thing, we will explain them to you as much as we can". Between the raging case of ADHD and my ongoing other disability issues I don't always have the ability to spend time on extensive explanations, but I do at least always have the willingness! (And a lot of our long-time users and volunteers will often know the answer even if I'm not around and answer it for you anyway!)

[igenlode]

Ah, I'm glad it wasn't a problem at my end, as I had assumed...

[badly_knitted]

Thanks for the update, I weas a little worried when it happened to me (based in the UK) a couple of nights ago, but waiting a few minutes did resolve things. Now I know what's going on I won't be worried if it happens again. 😊

[stefanyeah]

Thanks for the update. I was a little worried yesternight that I might have triggered some security measures as I was updating lots of old fic to their own space all day. 😅

[denise]

I promise it's not you! Trust me, no actual person can achieve the level of garbage we're talking about through normal site usage without automation.

[stefanyeah]

Glad to know. <3

[dantesspirit]

That explains the issues I was having. I chalked it up to my internet provider and tried again in the morning.

[scintilla72]

So in other words, you won't stop trying to make Backend Fetch happen?

[denise]

My wife earnestly assures me that my backend is fetching enough already.

[weofodthignen]

Thanks forletting us know :-) That was kind of what I figured when I was informed the guru had problems meditating.

[wallwalker]

It makes sense. Unfortunately, so does all of the trash that the internet is getting. Thanks for working so hard to keep it away from here.

[erulissedances]

Thanks for the update and all of the work you all put in.

- Erulisse (one L)

[theradicalchild]

Alright, glad you all are working on it.