dw_maintenance | slowness/downtime

I have no clue what this means, but it's good to know you guys are on the case. *g*

*points up*

That goes for me, too!

Thirded.

I will take a stab at making it approachable!

Think of it like having plates at a dinner party. You can put food on the plate, serve someone, and they eat it. Then you wash the plate and you can re-use it. That's roughly how Dreamwidth works, except we cycle through plates very, very quickly so the site runs on a small number of them (30 or so).

(Plates are the metaphor for Apache workers. A single worker is a thing that processes an incoming requests and generates a response, one at a time.)

Anyway, the community we found causes the site to break in a certain way that means the plate fills up ... but never gets emptied. Eventually, we run out of plates, and the line of people who want to use the site backs up out of the door.

We figured out what was causing the plates not to empty and put a band-aid on it so everyone can still eat.

Okay. Darnit. Now I'm hungry.

We figured out what was causing the plates not to empty and put a band-aid on it so everyone can still eat

First time I've ever heard of someone putting a band-aid on a full plate. *giggles*

Right. Dinner time...

Now you just need to make that into a video featuring Reactor-Kun.

Thank you for the explanation! *gets dinner ready*

(I now have a mental image of you lot all clearing dinner plates from a big banquet table. Seated around it are Geronimo, Chief Cochise, Mary Kim Titla, Jay Tavare, etc, etc. I'm sure this is not really how this stuff works, but it's a fun image all the same.)

Thanks for fixing this!

Of course, this analogy makes me think of the dining philosophers problem.

This is the best metaphor ever.

I just had the weirdest mental picture on reading that post.

Thanks for letting us know what's up, though. You guys are made of awesome!

Thanks! This issue was making me re-think getting a seed account, but knowing you guys are on top of it is good.

Not wanting to sound too rude, but a minor glitch has you reconsidering a seed account? No IT system in the world is perfect; not even Google, with all its resources has 100% uptime.

I'd understand this comment more if DW experienced significant outages (i.e. intermittent service for days at a time), but this is patently not the case. Also, the DW perps are fantastic at explaining issues as soon as they know what's going on. Please try and develop some real-world expectations, with the understanding that not one internet service is 100% available to all users all the time.

Please don't worry about other people's opinions, really! Everybody has the right to hold whatever opinion they want about DW, and particularly about whether or not they're comfortable buying a seed account. Not everybody is comfortable spending that kind of money for a service unless they're 100% convinced that it's stable and reliable, and downtime looks bad to people no matter what the reason is.

I admit it looked pretty bad to me, especially coming from LJ's extended DDOS! But the post from Mark pretty much reassured me that LJ and DW do things differently, which is one of the things that prompted my move _here_ rather than elsewhere, in the first place. ^_^

Let me just say here that I have been watching DW ever since its conception and launch, and I'm REALLY happy with what you guys have been doing, and that I am choosing to buy a seed account to support DW because I believe in the sort of transparency you bring. ♥

Edited 2011-04-25 15:03 (UTC)

I'm new to DW--I moved over during the Russian DDOS when LJ was pretty much inaccessible. It was INCREDIBLY frustrating not being able to get things done when I needed to do them. So when DW was timing out, timing out, and timing out again repeatedly, consistently, and over a few hours (I started trying before 9am on Sunday morning and it persisted until almost noon (as you can see from my timestamp) I was pretty frustrated.

Look at it my way. I announce to my LJ friends list that I am moving to DW, and half of my friends list says, "Oh, I'm sorry to hear that, I guess I won't be commenting on your blog as much as I used to." And then I get to DW, I pay $200 to buy 2000 points for a seed account on Sunday morning, and then BOOM, I get hit by the outages. What would you think in my shoes?

Also, I don't know about you, but my real-world expectations are _high_ when I am paying money. (I don't pay Google anything, and I use Google even more than I ever used LJ or DW, as they host all three of my e-mails and other cloud things.) If it does not work, the company hears from me. And if I cannot get to them, or if I talk and no one listens, I leave and I don't come back as a customer.

Edited ( ) 2011-04-25 15:12 (UTC)

Well...since LJ is fully functioning again and has given all users free paid time as a apology for the down time (which was definitely not their fault), you could have been...I don't know. Patient?

Please be careful when you're commenting to someone else in an official DW community! This comment is pretty close to being a personal attack, and we ask that people try to stay respectful of each other in official communities.

Anyway, everybody has different perspectives and opinions and expectations, and that's okay. We'll live up to some of them and fall down on others, and all we can do is hold to our guiding principles, let people know what's going on, and fix things when they break.

Thanks, Denise. I really do admire DW, and I really respect what you and your team have done and continue to do. Honestly, if I hadn't just seen your comment, I might have gone ahead and asked for the refund, but I want to support what you guys do. I'll just... stay away from zealots in the future and avoid posting here again.

LJ isn't selling seed accounts, DW is. Anyway, why are people jumping on me for thinking this? Does DW have some kind of ideal blogger profile I don't fit? If I'm not welcome, give me a refund on the 2000 points ($200) I paid and I'll leave.

ETA: thought better of what I said. Bad day at work, tired, grumpy, can't afford the $200 but doing so anyway. Forget I said anything, and I'd like someone to please screen or freeze this thread so no one else jumps on me? Thanks.

Edited 2011-04-27 00:52 (UTC)

I've frozen it. :)

Wondered what the heck as your near instant response time has spoiled me rotten. Thanks for the quick info and fix.

Heh, now I'm wondering which community it was...

A very small one with five members and many, many comments arranged in such a pattern as to cause this. It wasn't on purpose, so I'm not going to publicly out them. :)

That's fair! I was just worried it might've been one of the ones I run and I didn't notice. *g* Thanks.

Definitely agree on the not-outing them. Is it possible to say what they were doing to cause this without outing them? In a general way, maybe?

Basically, so as to warn us so we don't do it, I'm guessing?

Yup. Although, if they're fixing the code so that it won't happen again, that may not be necessary. However, might it also screw up, say, LJ? Or other sites that we might be using?

You won't run into it -- we've identified the problem with it and are working up a fix. It won't even be relevant soon. :)

Awesome! I'll stop worrying, then. :)

As soon as the final patch to fix the problem is pushed out, sure!

Unforeseen enthusiastic nesting in Spring?

XD
^_^b

Ah, so it's not that we have an ONTD clone on our hands, then. :D

One is ... curious as to the exact details of the problem, and quite certain that the Apache project would be interested to hear them too.

It was definitely an inefficiency problem in our code, not in Apache. We're waiting for a less hacky fix, and to see if LJ is vulnerable too, before we give much more detail, since if it were done maliciously instead of accidentally on a site that was vulnerable, it could be very very bad.

And that is, I believe, exactly the correct way to handle it. Kudos to you. Again. :)

Thank you for working so hard.

Thanks for fixing...washing...band-aiding... ...umm.

Thanks for doing a good job!

Best,

–Nic

Thank you for your hard work

fu and

mark!

I'm curious as too how a small comm manages to hog all the DW plates though. That seems like an odd one :D

My personal guess: a roleplaying game or something with very, very long comment threads. From the comments I see we are likely to get a description of the type of thing it was after there is a permanent fix, yay!

Edited 2011-04-25 02:12 (UTC)

Yes --

It was a small RP community. They had gotten a post up to 1400 comments, which is really nothing, but they were arranged in a chain. Reply, reply, reply, reply. So instead of the normal view of things -- a wide tree -- we had a very, very deep tree.

The part of the code that checks to see if you are subscribed to a thread walks the tree from where you are up to the top to see if you're subscribed to any comment in the chain. Normally, this is simple -- you walk a few comments up and you're at the root and you're done.

In this case, it had to walk 1400 comments. The code was done in such a way that it had to do this frequently, and it caused an explosion of work on Apache, taking tons of memory and CPU. Ouch.

fu fixed it temporarily with some better caching to make sure we don't have to recompute things we already know. We may come up with a better way to do it, but it works for now.

Aaaaaaaanyway. I know I didn't break that down into plate analogies, but I have to run. Maybe someone else can, if needed, and if not, I'll do it later if I remember. ^^;

Oooh, oooh, I'll try!

So, whenever you and your friends go out to dinner, you all share your food by picking up your plates and handing them around the table so everyone can get a taste. When you've got four people at the table, that's no problem, you can make one circuit of the table really quickly and finish your meal and the plates can get back to the kitchen to get washed and reused.

But, say you've got a really big party. And everyone wants a taste of everyone's dinner still. So, you pass the plates around the table just like you're used to doing, but since there are so many of you, it takes the plate forever to make its way around, and by the time it's only halfway around your group, the restaurant says, "Um, guys, can you speed this up? We totally need the plates."

Ooo, I like yours.

Dodgy analogies are a specialty of mine!

So there's this person building a staircase who needs to be served as well. One of the workers runs up a plate to the guy who is at the top of the staircase, going step by step up. It takes far longer for that plate to get there and back again than for someone in the main dining hall, and then it might not even get washed. And the staircase keeps increasing in length, but the guy on the staircase still needs to get served, so it takes longer and longer and the person doing the plate-running starts to get very tired out.

This is a situation for a bucket on a pulley.

So the loop isn't really infinite, just a very long one, fair enough. :)

Yeah. When I made the original post, I thought it was an infinite loop. It turns out it's not, it's just a really bad tree traversal problem.

Excellent metaphor. And thanks for the relatively quick fix, as well!

Many thanks, guys. I'd gotten used to the fast response time, and the whole "falling off a cliff" thing was really unnerving.

I noticed things were a bit funny at DW today. Not just the time to load pages, but some of my photographs were wildly screwed up. I started to replace some of them, only to find that when I went to a separate window, they were all fine. Yet every time I'd refreshed the first window, the same photos were massively pixellated/corrupted. Hope the problems are sorted, or sorted soon.

Thank you! I really *really* appreciate you letting us know what the problem was, as well as patching it!

No worries- I figured it was something higher up, or something with the local 'net connection.

Thank you for pouncing on it!

I'm just happy to hear that the issue has been found, is being fixed, and is not a DDoS.

Ditto! That was my concern earlier when I couldn't load the site at all.

Thanks for keeping us informed!

ETA: Also seems Facebook is down (or is that just me)?

Edited 2011-04-24 18:32 (UTC)

I'm glad I'm not the only one who had that particular panicked thought. *g*

http://downrightnow.com/facebook will give you any recent history of problems and http://www.downforeveryoneorjustme.com/facebook.com will tell you if problems are local.

Though it takes more for downforeveryoneorjustme to register a problem.

Both work with other services; LJ looked terrible during the DDOS. You could see the drop followed by a climb back up and then another drop.

Oh, wow, best links ever. Thank you! *bookmarks*

Hear, hear. I only had trouble for a little while but LJ was also very slow for me too so I couldn't help but think of the worst. Then I saw the message on

dreamwidth. :) Thank you Mark and Fu for the quick fix and for keeping us informed. You both rock!

*offtopic* That is the first time I've seen the little twitterbird userhead! That's so cool. &dreamwidth;

I hadn't even noticed a problem. (The joys of ping rates over 1000ms.) As always, thank you for keeping users utd regarding issues that could affect us.

Thank you for the explanation. I did notice a little slowness earlier, but put it down to general network responses. I should have realised there was some sort of problem as DW is one of the most responsive sites I use :-)

Well done for the rapid diagnosis and the temporary workaround!

Thanks for letting us know! I'd been wondering what was going on.

I wonder if this explains why I randomly get timeouts sometimes during what I assume are probably peak times. Never takes more than a single reload to fix it, but happens fairly regularly. I'd say at least once a day I get this. Never able to really track it down, because it always works again immediately.

Interesting, though. I'll have to wait and see what the fix looks like, to see what the heck was going on.

Thanks for the explanation! I only thought of DDoS in retrospect - when I was having a lot of trouble with it earlier, it was mixed up with my broadband going up and down the way it does sometimes (a few weeks ago I reckoned it was the wet weather, what's it upset about now?).

Edited (fix punctuation) 2011-04-25 10:34 (UTC)

:D Isn't it fun when people find new and exciting ways to break stuff? The spice of life!

I AM SO SORRY. *HIDES* I just wanted to expand the thread! So I could save it! And not have to open it in tiny tiny pieces when I wanted to reread it! I had no idea I was causing ACTUAL PROBLEMS adjskg;dsjgkl aaaaaaaaaaaaaah.
*hides head in hands*

I AM SO SO SORRY.

It's okay! Don't worry about it, really -- you found a problem that we could fix. :) I'm just sorry we had to whack the entry temporarily before we could get the problem fixed.

(ps: even though we've put in a fix, and the fix is live now, you still might want to break a thread every 100 or 200 comments just because.)

/o\ SO EMBARRASSMENT. (Meep! I didn't even notice that! I was asleep! Aaaahhh. Luckily I saved what I'd got so far, I guess. ._.)

(Yeaaah, I'm getting that idea. /o\ I'm sorry!)

*g* It's okay! Really! Really seriously. It's something we wouldn't have found without you, and it's something that should be fixed. So it's good, in a way.

(Obviously it would've been better without the downtime. But still. *G*)

(We'll unsuspend the entry as soon as we're sure the problem is fixed!)

^^;;;;;; Still!

(I didn't even know there was any downtime! My internet was buggy yesterday so EVERYTHING was slow. ...unless I was causing my own internet problems. *facepaaalm* I totally was, wasn't I?)

(Yay! I'm sorry!)

I will say this, since this is one of those things that more voices may help with:

You did not cause a problem; you FOUND one. Finding the bits that are goofy so that they can be fixed = a GOOD thing.

*hands over BugFinder button*

I - thanks. I do feel a bit better. Still a complete moron, but. Yeah.

Hey, look at it this way -- before, we had a problem and we didn't know we had a problem. Now we have a fixed problem, and you're the reason it got found to be fixed!

I count that as a net win. :)

Seriously! We do not blame you guys! We do not blame anybody. The problem was there, whether or not we knew about it. Whenever you're dealing with edge cases or people using the site in a way that is way outside the way that most other people use it, there's always the chance for bugs to crop up, because nobody has ever tested those particular combinations. It's why there are always bugs that don't show up until we push code live, because our users use the site in ways we never would have dreamed, much less tested.

So do not feel like a moron! Feel like somebody who just contributed to making DW better for everybody, because you totally did. :)

...thanks. ._.;;
Uh. Good thing it's Three Weeks, so my shame and related Issues aren't allowed to drive me away from the site for a month or so to recover? I have obligations. ^^;;

And, uh. I was going to buy a seed account anyway, 'cause apart from this whole breaking it thing I do love this site like burning - but instead of waiting to see if I can nab one I'm gonna go buy the DW points now, so even if I don't get one you guys get my support.

Noooo! Look, hi, I'm not involved in dreamwidth dev, but finding bugs is a GOOD THING, you are awesome, please stay, you have helped make it Better For Everyone.

^^;;;; It is getting funnier and less crushingly embarrassing as time goes on?
And yeah, Three Weeks obligations, so. Around anyway.

Another voice, from another fellow user and long-time bugfinder: Finding problems is great! It's especially good when it's a low-traffic day to start with (this being Easter Sunday, there are a lot of people doing non-internet things, and while I don't know how DW users stack up against the internet-use-in-general profile, it's got to be somewhat less than typical weekend traffic), during a low-traffic time of the week (all the stats I've heard have said it's lower-traffic on weekends in general), not during the seed account sale, and found by someone friendly rather than someone with any sort of malicious intent.

In the IRC channels where many of the developers hang out (I am not a developer, but I make bad puns with many of them) there is a bot that lets the whole chat know when a new public bug has been filed in the bug-tracker. Occasionally, not as often these days because there aren't as many good ones, but sometimes, there is cheering, actual cheering, when a new bug is filed. This is because the bug getting filed means that there is enough information about a problem to formally enter it in the system, and because filing the bug is the first step to getting it fixed, and having it fixed means that no one else will ever be able to (deliberately or accidentally) do the same thing and cause the same problem ever again.

This is definitely a \o/ YAAAAAY!! \o/ bug.

I don't know if you've ever had the experience when writing fic, where there's something, and you're working on making it the best writing you can possibly manage, and you hand it off to a beta, and the beta points out something that screws everything up, but when you fix it, it is going to make things more awesome than you dreamed was possible? And you're a little sorry that you made a mistake, but you know no writer's perfect so it's not like you're the only one who's done something similar, and it's going to be a lot of work to fix, but once you fix it, it's going to be better than you had thought you could write, and it's exciting and the writing to fix it will be the fun kind of challenge? I would compare some bug report experiences to that. And in this analogy, you would be the beta who pointed it out, not the writer who made the mistake or the writer who's going to have to fix it.

Meep! #._.#

Um. If you say so? And yeah, at least it was this weekend, not next. That would have been. Um. Bad. Very very bad.

I will try and take your words to heart!

♥

Once before open beta, I managed to partially break the importer because I'd copied an email into my LJ and between one thing and another LJ told the importer that it was sending an email to import, so the entry page confused the importer a whole, whole lot.

And then the devs fixed it and everything was ok again!

That was a *hilarious* bug.

That was also a "why is Mark's hair all over the floor?" bug. I'm bald now. Thanks, buggy Perl modules!

It was awesome, especially after Mark figured out what was going on. To add to the fun, I'd made it private, so he very apologetically had to ask if he might wade through my import data, and then asked me if there was a post between here and there...

Nah, seriously.

i'm currently spending a lot of time playing the alpha2 release of FreeCol, and trying to find bugs. The obvious ones I don't need to report ("um, guys, why aren't my horses breeding/ That's kinda important"). So I'm playing with some batshit insane strategies to see if anything weird comes up that the Devs have never seen before. Thus far, I've found one thing.

The lead Dev has thanked me profusely (because, seriously, it took a lot of weird setup to find this thing, although I wasn't looking for that specifically I was just trying to break the game).

Now, it's fixed in trunk so when 0.10.beta comes out the game might actually be fully playable.

Dreamwidth is still in beta, we're still testing the servers, the code, the whatever.

You have found a bug. One they hadn't seen before, at all, one that shouldn't exist.

This is a Good Thing.

Because now thye can fix this bug, and possibly make sure that a) you can't do it again but more importantly, b) other people doing something similar but different don't do it again.

Congratulations, you have broken the site in a way they can fix. Really, that's a good thing to do.

^^;;;

As someone else who uses DW for tiny little RP things (and is currently too lazy to sign out of an RP journal >>), let me just say: Heck, I'm not even mad; that's amazing.

♥

^^;;;;;;;;;;;;;;

Having skimmed through the comments to figure out what was going on, I also have to thank you. I'm an RPer who loves long threads and haaaaaaaates breaking things up into multiple little threads. If it hadn't been you, it probably would have been me sooner or later.

Don't get down on yourself! Now it's fixed and we can all play in the sunshine and rainbows (...and dinner parties). :D

^^;;;;;; Well, at if nothing else I'll always remember to break up my comment thread now, aha. *facepaaaalm*

Uh. I'm just here to say that you guys are awesome and I love you. Don't ever change. <3

Seconded!

Thanks for letting us know. Outages and problems happen no matter how well you plan - but I would commend the _response_ - not just the patching/fixing it (I'd assume everybody does that) but the part where you tell us what is going on (and one hopes that this is not a greater vulnerability of the platform).

Yes :-)

To be honest, DW is so awesome, fast, and reliable that I assumed the problem was on my end anyway. :D

I grieve with you for your dead Apache workers!

slowness/downtime

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

(frozen comment) no subject

(frozen comment) no subject

(frozen comment) no subject

(frozen comment) no subject

(frozen comment) no subject

(frozen comment) no subject

(frozen comment) no subject

(frozen comment) no subject

(frozen comment) no subject

(frozen comment) no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject