diagnosing site slowness

[mark]

Hi all --

Hello! I'm back from the wilderness of Alaska, and I'm now helping alierak, fu, and denise to resolve the recent issues Dreamwidth has been experiencing.

I'm going to be working on things over the next hour or two, so you may see a bit of a blip as I redirect traffic and make some changes. Please feel free to let me know here if anything goes awry badly, but it should be pretty smooth.

Thanks for all of your patience through this. Slowness is sad, we'll do what we can to nip it in the bud. :)

Edit: I've made a change to how we route static traffic to reduce load on our load balancer. If you notice anything wrong with images or embedding (or anything else!), please let me know.

Comments

[biscuit1001]

You guys ROCK!

[jumpuphigh]

Slowness is sad

This made me laugh so hard.

[goldlure]

thanks for your job)

[weaverbird]

Thank you for all you do for us!

"Slowness is sad..."
HEE!

[rosefox]

I just assumed it was the flood of people coming over from LJ.

Happy site-fixing!

[denise]

The load helped us to highlight the problem -- it definitely wouldn't have been noticeable without the load -- but the load wasn't the only factor contributing! It's definitely given us a good window into the problems that we can fix. :)

[bob]

nothing like real load to really stress test a system and find the painful bits.

simulated traffic isnt really the same :)

[denise]

Exactly! And it's always painful when you've dealt with this level of load before, so you get to sit there and go, okay, what's changed since then?

[risha]

But that's my favorite kind of debugging! That way you have a starting place!

(The deepest pit of debugging hell, of course, is the major issue that spontaneously fixes itself.)

[denise]

But only after you've spent dozens of hours tearing out your hair, of course!

[risha]

And then you bitterly wish that it was still broken!


(Actual logged defect I received last Friday: "Definitions page on SUMEOB is working".)

[brownbetty]

Aaahahah, YES. WE HATES IT.

[kyrielle]

Oh man I loathe those.

My second favorite is intermittent, rarely-occurring, but major issues (stack trashes, a perennial win). But when they spontaneously go away...and you're not sure they'll STAY gone....

Had a co-worker who put in logging to trap an elusive problem. Problem, which had been happening about once a week, has not been seen in the two months since logging was placed. Co-worker: not really very pleased by this.

[denise]

There is a special hell for Schroedingbugs that seem to go away permanently when you put in code to observe them. :)

[doro]

LOL

Ow, my tummy muscles wore out laughing.

*hugs*

[archangelbeth]

This is like my car, which had its "check engine" light on, and was flashing its cruise control, for a week -- until I got into it to drive it over for its appointment to get checked out. I turned it on, and there were no blinkenlights complaining. It went over sweetly to the dealership (they're a good dealership, unlike some of the others around), allowing cruise control to be activated, and was just as sweet as if extra-premium wouldn't melt in her mouth.

(Then they checked her over and on the way back from that, she decided to have one of her oxy sensors start failing, so that wound up having to be replaced.)

[dragonfly]

Are there really loads of people coming over from LJ? I thought you were being ironic.

[rosefox]

I put 17 invite codes up in dw_codeshare and they were gone in less than a day.

[dragonfly]

Cool. I think I'll go put some there, then.

[holyschist]

Account creation spiked really hard last week, and likely came with the usual attempts to import, among other things.

[risha]

So that must have been fun to walk into post-vacation. :)

[arrghigiveup]

Lol, I was thinking this too. On the bright side, it's good to feel needed, I suppose? XD

[ninetydegrees]

Thanks for the update, Mark. Hope you had a nice time in Alaska!

[brownbetty]

Good luck, and godspeed!

[ficwriter1966]

Thanks for your hard work, Mark! Hope you had a great time in Alaska!

[delphia2000]

Thank you, Mark....and thank you for spending your tourist dollars here in the Great Land!

[spb_zaika]

Good luck! Удачи!

[owl]

It is a well-established fact that whenever the tech lead or head sysadmin goes on holiday, everything will proceed to go to crap ;-)

Good luck! Memory leaks are killer, we've had one in production for going on two years (we turn it off and on again every morning....)

[cheezdanish]

The hard work is greatly appreciated, and thank you!

[jumpingrat]

Thank you guys, transition from LJ works pretty well and i like this place even more. Good luck!

[hendrikboom]

When you've finally tracked it down and fixed it, I'd love to see a technical discussion on how you managed to find it!

-- hendrik

embedding in community comment

[frith]

You asked about embedding snafus. 8^) In a comment in ponyville_trot, here, the last one on the page: http://ponyville-trot.dreamwidth.org/6407.html#comments I attempted to post an embedded Youtube video (in the old version of the script, the one that does not require that Javascript be enabled for the embed to display). As you can probably see, no embedded video was displayed. I searched the FAQ for 'embedded' and 'youtube' without success, nor was I able to find anything about enabling embeds in comments for the community. I thought that it might be something cut to improve performance on the site while you wait to update the Perl but there was no mention of disabling embeds in the latest dw_news either. So, since you asked, I'm reporting it here instead of opening a ticket in Support... If you like, I could still open a ticket in Support. Here's that embed again, I'm expecting it to fail to display here too: