Tonight's site interruptions

[mark]

Hi all,

First of all, sorry for the trouble tonight -- I know a lot of you had issues accessing Dreamwidth tonight.

TL;DR: the issue is fixed and service is restored. The issue was one that we solved by changing the configuration of the web server software we use (Apache).

The basic description of the issue is that we believe that all of our Apache workers were being tied up doing keep-alive to clients. There is a 5 second timeout after you request a page where we wait for you to request another -- and if you don't, we close the connection.

It's possible for there to be enough users that we run out of connections, and then you have to basically wait in line for another connection. This is usually very fast, but with lots of people trying to use Dreamwidth tonight, we ran out of free connections. This caused a cascading problem though -- because our upstream load balancer _also_ needs connections in order to do health checks to ask if the web servers are doing OK.

So, we started seeing oscillating health check failures. Web servers would start to be considered failed (because the load balancer couldn't ask it how it was doing) and that caused us to have even fewer connections to be available, which just exacerbated the problem.

It was hard to track down because this really shouldn't be the case -- the load balancer should be multiplexing requests to different backend connections. We need to do some more investigation to understand why we were seeing this behavior. At any rate, we resolved it by disabling keep-alive.

Again, we're sorry for the interruption of service, and that we didn't tweet faster. That's on me, I forgot to update over there while we were working on the issue.

Big thanks to jennifer and alierak for the help here! Ultimately, it was alierak's suggestion that pointed us in the right direction and got the issue solved.

-Mark

Comments

[egret]

Thank you for getting it running again!

[darkoshi]

Thank you! I'd been wondering if it was a problem on my side or not.

[bemused_writer]

Thank you for letting us know and for getting DW up and running again!

[coinneachf]

Load balancer healthcheck issues are waaaay too familiar to me. Good on yer.

[weofodthignen]

Thank you for the fix and for being open about it (and trying to explain it to non-techies). I'd thought it was at my end.

[mc776]

Y'all are amazing at both fixing and documenting what goes on around here. Thank you.

[themagdalen]

https://www.isitdownrightnow.com/

[lunabee34]

Thank you so much for fixing the problem!

[dewline]

Thanks for clearing it up for us.

Also, mildly and perversely grateful because I already needed to call it a night when the problems started affecting my connection to Dreamwidth. Good timing in my case, and I recognize that such was not the case for all my Dreamwidth "neighbours".

[darkoshi]

Oh, that is a useful site. I'm bookmarking it.

[primwood]

Thanks, Mark, Jennifer and Alierak!

[havocthecat]

Thanks for all your hard work!

[numb3r_5ev3n]

Thanks for the explanation and the swift recovery!

[kore]

Thank you for being so transparent and fast about it! I love Dreamwidth.

[isis]

Thanks for getting on it and fixing it, and also for coming here to tell us about the problem. Dreamwidth is awesome!

[paynesgrey]

Thanks for all your hard work!

[renay]

Thanks for the quick response and explanation. Y'all are great.

[tozka]

Thanks for the quick fix!

[monanotlisa]

Good to hear -- thanks, Mark!

[sistawendy]

From one erstwhile ops person to another, I salute you. Your load balancer wouldn't happen to be in AWS, would it? (I know, you shouldn't answer that.) My co-workers observed some annoying behavior in the distant past whereby AWS LBs would just point a firehose at one server until it fell over, then move on to the next one instead of proper round robin.

[sine_nomine]

Tried that last night. It insisted the site was up. While I was getting 504 messages.

Pobody's nefect?

[claidheamhmor]

Thank you for the clear explanation. Load balancers can be fun!

[mudousetsuna]

Would this error have anything to do with why LJ Juggler extension on firefox no longer works? It keeps saying that I probably have typoed the username or password, and I know I haven't.

[cesy]

You'll need this one: https://dw-dev.dreamwidth.org/221358.html - the main announcement was https://dw-maintenance.dreamwidth.org/81112.html

[mudousetsuna]

Oh I see now, thank you very much!