Tonight's intermittent 504s...

[denise]

...have been due to a very badly behaved bot hitting us ridiculously hard. We blocked it a few times, but it kept trying to work around the block. We've now gone for the nuclear blocking option, which we hope will keep the problem from returning, but if the problems start up again, we'll continue to fix them as soon as we can!

Comments

[shanghaied]

If there were only some way to tell the bot(s) that it's LJ they're (probably) looking for... :P

[marahmarie]

When there are these outages (since they've been a thing a few times) what is the preferred way for us to report them? Does DW staff want us to leave Support requests (if we can get through the time-outs - I had to re-load the Support request page quite a few times to write the request) or to contact you on Twitter (I don't have a Twitter) or in some other way (email)? I'm curious because in that sort of situation it seems whatever way is fastest and actually works might be the best, but I don't want to leave any site feedback in places it isn't already welcome.

(Also, slightly O/T: the read-only version of Dreamwidth! I did not know it existed. The refresh button on that page, and the offer to refresh the page for you when it becomes available again, are truly some great things. Never saw that before tonight.)

[mirrorofsmoke]

Oh nooooo! Evil bots of doom!

[paw_prints]

Ohhh I did wonder what was happening! I thought my internet or laptop must have gremlins.

Hope it's got the hint and has gone away now.

[artem772]

It's revenge of Russians for LJ )

[redhedlvr2]

My thoughts exactly.

[denise]

Someone's badly programmed homework assignment, actually!

[denise]

Someone's homework assignment, actually, as far as we can tell! They just forgot about the "rate limiting" part...

[shanghaied]

Hee! Quelle drollerie :D

[denise]

Generally, if the site's completely inaccessible, we already know there's a problem -- we have automated alerting that notifies us of problems and most alerts get addressed before it proceeds to something that y'all can notice. If something doesn't get fixed very quickly, it's either due to high traffic/load and we're working on it or it's happening at a time when nobody's been awake/available/in front of a computer to handle the alert yet and we'll get it as soon as we can. (We don't, alas, have the resources for 24/7 ops coverage, and despite our best attempts at maximizing the coverage we do have, there's always going to be times when someone doesn't wake up for an alert.)

The real problem happens when there's a degradation of service that doesn't hit the point of being severe enough to trip any of the automated alerts, but every time that happens, we poke at things to see how we can tune the alerts to be more sensitive to legit problems without triggering when we're just a little busier than usual.

[marahmarie]

Thanks for the information; that's all very good to know.

[denise]

And we do try to be diligent about dropping a quick "we know about the problem, we're working on it" on Twitter as soon as we actively start working on an issue, but we don't always remember! 

[marahmarie]

Oh, you mean it actually was! Someone referenced a college in the Twitter post last night *checks Twitter*...yes, this one:

https://twitter.com/dreamwidth/status/861051012542541824

That must have been one heck of a homework assignment.... Go Penn!

[marahmarie]

That's also good to know (and I always check DW's Twitter as soon as things start acting wonky here, so at least now I know that's the right way to keep track). Thanks!

[madfilkentist]

So the Net has reached the point where people can launch DoS attacks purely by accident.

[gingicat]

I was worried it was something like that. Go you!

[prototype_infiltraitor]

Oh, sure.

It's always the bot's fault.

Never the programmer.

[vitus_wagner]

Google for "Morris Worm". Net reached this point about thirty years ago.

[denise]

(this comment totally cracked me up)

[denise]

oh lord it really has been thirty years hasn't it

this icon has never been more apt

[chanter1944]

Oh my goodness, it actually was? *sporfle!* And here was me suspecting it had to do with LJ, today's election in France, and a certain deliberately-unnamed nation... On the actually extant hand, that is one heck of a frakup! Somebody's either going to get an earful from their professor come Monday, or else join said professor in mopping egg off their face--oh dear!

*giggles all the way back to her own journal*

Re: this icon has never been more apt

[denise]

No, it really was a misconfigured bot from someone's CS homework! I eventually found the assignment and realized it's due Monday, which makes me think the botmaster took out all the rate-limiting to get the crawling done before then...

Re: this icon has never been more apt

[lysanatt]

OMG, if it wasn't so annoying for you it'd have been hilarious. Nah, it IS hilarious. And thank you for solving it so quick.

Re: this icon has never been more apt

[denise]

It was kind of funny even when we were in the middle of it, yeah. (And it would've been fixed even faster if it hadn't been late enough at night that I couldn't wake anybody up at first! At least now I have access to the thing that will let me fix it in the future if I'm the only one awake...)

[autopope]

* Shuffles embarrassedly *

I did that, once. (But because I did it in 1993, the main consequence was the invention of the robots.txt file, rather than a DDoS on a social platform; the consequence of saturating a server on a 14.4K leased line with GET requests from a box with access to a humongous 64Kbps of bandwidth! The idea of needing an exponential back-off didn't even occur to me. I mean, who would run a web server on a 14.4K line?)