slowness/downtime

[mark]

Hi all,

In the past 24 hours we've had some periods of slowness and outright downtime. fu and I have managed to track it down to a single community that has a comment thread that causes our Apache workers to go into an infinite loop. After enough refreshes, all of our workers are dead, and the site no longer responds.

We're going to start looking into fixing the code so that this doesn't happen. Meanwhile, we're going to work with this community to make sure that they can't break the site while we get an actual, proper code fix working.

Thank you all for your patience as we sort this out.

Comments

[rydra_wong]

I have no clue what this means, but it's good to know you guys are on the case. *g*

[weaverbird]

*points up*

That goes for me, too!

[sharpiefan]

I just had the weirdest mental picture on reading that post.

Thanks for letting us know what's up, though. You guys are made of awesome!

(frozen comment)

[februaryfour]

Thanks! This issue was making me re-think getting a seed account, but knowing you guys are on top of it is good.

(frozen comment)

[trixtah]

Not wanting to sound too rude, but a minor glitch has you reconsidering a seed account? No IT system in the world is perfect; not even Google, with all its resources has 100% uptime.

I'd understand this comment more if DW experienced significant outages (i.e. intermittent service for days at a time), but this is patently not the case. Also, the DW perps are fantastic at explaining issues as soon as they know what's going on. Please try and develop some real-world expectations, with the understanding that not one internet service is 100% available to all users all the time.

[yanagi_wa]

Wondered what the heck as your near instant response time has spoiled me rotten. Thanks for the quick info and fix.

[trialia]

Heh, now I'm wondering which community it was...

[mark]

A very small one with five members and many, many comments arranged in such a pattern as to cause this. It wasn't on purpose, so I'm not going to publicly out them. :)

[sidneyfireblood]

Thank you for working so hard.

[needled_ink_1975]

Thanks for fixing...washing...band-aiding... ...umm.

Thanks for doing a good job!

Best,

–Nic

[poulpette]

Thank you for your hard work fu and mark!

I'm curious as too how a small comm manages to hog all the DW plates though. That seems like an odd one :D

[azurelunatic]

My personal guess: a roleplaying game or something with very, very long comment threads. From the comments I see we are likely to get a description of the type of thing it was after there is a permanent fix, yay!

[epershand]

Excellent metaphor. And thanks for the relatively quick fix, as well!

[nightdog_barks]

Many thanks, guys. I'd gotten used to the fast response time, and the whole "falling off a cliff" thing was really unnerving.

[bellakara]

I noticed things were a bit funny at DW today. Not just the time to load pages, but some of my photographs were wildly screwed up. I started to replace some of them, only to find that when I went to a separate window, they were all fine. Yet every time I'd refreshed the first window, the same photos were massively pixellated/corrupted. Hope the problems are sorted, or sorted soon.

[giglet]

Thank you! I really *really* appreciate you letting us know what the problem was, as well as patching it!

[jecook]

No worries- I figured it was something higher up, or something with the local 'net connection.

Thank you for pouncing on it!

[blackmare]

I'm just happy to hear that the issue has been found, is being fixed, and is not a DDoS.

[jadedmusings]

Ditto! That was my concern earlier when I couldn't load the site at all.

Thanks for keeping us informed!

ETA: Also seems Facebook is down (or is that just me)?

[jumpuphigh]

I hadn't even noticed a problem. (The joys of ping rates over 1000ms.) As always, thank you for keeping users utd regarding issues that could affect us.

[rpdom]

Thank you for the explanation. I did notice a little slowness earlier, but put it down to general network responses. I should have realised there was some sort of problem as DW is one of the most responsive sites I use :-)

Well done for the rapid diagnosis and the temporary workaround!

[trobadora]

Thanks for letting us know! I'd been wondering what was going on.

[enotsola]

I wonder if this explains why I randomly get timeouts sometimes during what I assume are probably peak times. Never takes more than a single reload to fix it, but happens fairly regularly. I'd say at least once a day I get this. Never able to really track it down, because it always works again immediately.

Interesting, though. I'll have to wait and see what the fix looks like, to see what the heck was going on.

[susanreads]

Thanks for the explanation! I only thought of DDoS in retrospect - when I was having a lot of trouble with it earlier, it was mixed up with my broadband going up and down the way it does sometimes (a few weeks ago I reckoned it was the wet weather, what's it upset about now?).

[jadey]

:D Isn't it fun when people find new and exciting ways to break stuff? The spice of life!

[tyger]

I AM SO SORRY. *HIDES* I just wanted to expand the thread! So I could save it! And not have to open it in tiny tiny pieces when I wanted to reread it! I had no idea I was causing ACTUAL PROBLEMS adjskg;dsjgkl aaaaaaaaaaaaaah.
*hides head in hands*

I AM SO SO SORRY.

[denise]

It's okay! Don't worry about it, really -- you found a problem that we could fix. :) I'm just sorry we had to whack the entry temporarily before we could get the problem fixed.

(ps: even though we've put in a fix, and the fix is live now, you still might want to break a thread every 100 or 200 comments just because.)

[kitters]

Uh. I'm just here to say that you guys are awesome and I love you. Don't ever change. <3

[jeweledeyes]

Seconded!

[green_knight]

Thanks for letting us know. Outages and problems happen no matter how well you plan - but I would commend the _response_ - not just the patching/fixing it (I'd assume everybody does that) but the part where you tell us what is going on (and one hopes that this is not a greater vulnerability of the platform).

[kaberett]

Yes :-)

[eleanorjane]

To be honest, DW is so awesome, fast, and reliable that I assumed the problem was on my end anyway. :D

[gozer]

I grieve with you for your dead Apache workers!