mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
Mark Smith ([staff profile] mark) wrote in [site community profile] dw_maintenance2012-04-28 05:22 pm

Site slowness expected shortly

Hi all,

Summary: some unexpected but required maintenance is going to cause the site to slow down by some amount for the next hour or two.

Some details: We have run into an issue with one of our larger communities that requires us to make a change to the way we store comments in memcache. This means that our caches are all going to invalidate here in a few minutes when I roll the code out, so the databases are going to get very busy until we repopulate the caches.

Those of you who have been on LJ for a few years might remember when ONTD ran into an issue when they hit 16,777,215 comments and LJ had to do the same thing. I helped LJ with the code to fix the problem, but for some reason, never committed it on Dreamwidth. Now we have hit the same issue and I feel silly for having not done that, but here we are, and hence this maintenance.

Please let me know if you see any major problems, and again, sorry for the inconvenience. This really isn't a great time to be doing site maintenance.

Update: Congratulations to [community profile] sixwordstories for being the first community or journal to hit the 16.7 million comment limit in the database. :)

At any rate, currently the community is being migrated to a database where I fixed up the column widths so they won't have this limit anymore. It will take some time to move that much data, though. In the meantime the community will be in read-only mode.

I'm sorry that we hit this -- I really thought we had made this fix on Dreamwidth after LJ ran into it. It turns out we didn't, though, hence this situation. Alas.
subluxate: Sophia Bush leaning against a piano (Default)

[personal profile] subluxate 2012-04-29 12:39 am (UTC)(link)
It is what it is, regarding when you have to roll it out. Thank you for keeping us updated.
birke: (Default)

[personal profile] birke 2012-04-29 12:39 am (UTC)(link)
No prob.
0jack: Photograph of a heart formed from the negative space left by the arrangement of a rainbow of crayons. (All shades of love.)

[personal profile] 0jack 2012-04-29 12:41 am (UTC)(link)
Oo, I just want to know which community. *grins* *is bored, wants to read*

Good luck with this! :)
antilove: (I can't stop laughing)

[personal profile] antilove 2012-04-29 12:48 am (UTC)(link)
Thanks for the heads up.
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)

[personal profile] sophie 2012-04-29 12:52 am (UTC)(link)
The fact that a community on Dreamwidth is having the same issue as ONTD had makes me feel like little DW is growing up!

[edit: Of course, DW isn't so little now anyway. But still...]
Edited 2012-04-29 00:52 (UTC)
helens78: A sheep with the DW logo on it dreams of Dreamwidth. (dreamsheep: dw swirl)

[personal profile] helens78 2012-04-29 12:54 am (UTC)(link)
Hee! Thanks for the heads up; yay for getting the fix committed, and it's always so nice to be informed that it doesn't seem like a terribly inconvenient time at all. ^_^ (I think this is maybe a geek mindset thing; if we know WHY!! something is happening, it's so much easier to deal with it, no matter when! Or maybe that's just me.)
ariestess: (Default)

[personal profile] ariestess 2012-04-29 01:20 am (UTC)(link)
No worries. I'd rather know ahead of time and be able to plan for it where applicable.
mm_writes: Sheep go to heaven, goats go to hell (Default)

[personal profile] mm_writes 2012-04-29 01:37 am (UTC)(link)
Has DW really hit 16,777,215 comments in just one of the communities? Way to go! This site must be getting waaaaaay more traffic than I thought was probable.

Uh, sorry Mark, couldn't resist; but I *am* curious if your earlier statement about 2^24 comments in that comm and the numerical figure that tied up LJ are the same - again?
Edited 2012-04-29 01:38 (UTC)
rosalynian: (sorariku | we'll go together)

[personal profile] rosalynian 2012-04-29 03:05 am (UTC)(link)
Your openness with the userbase and letting us know what's up each step of the way is just the loveliest thing, thank you.



It's so nice just being allowed some looks behind-the-scenes of what you're doing; it really helps the site feel friendly and accessible. I'm so glad I and so many others migrated here!
heartequals: natasha from the avengers against an orange and yellow background (Default)

[personal profile] heartequals 2012-04-29 05:39 am (UTC)(link)
Round of applause for [community profile] sixwordstories! That's awesome!
jeweledeyes: Bender from Futurama groovin' old-fashioned style (Bender dance)

[personal profile] jeweledeyes 2012-04-29 05:43 am (UTC)(link)
If you have to have a problem, this seems like a good one to have, lol. Go DW and [community profile] sixwordstories!
zillah975: Detail of a sleeping cat from Pierre-Auguste Renoir's Sleeping Cat, 1862 (Default)

[personal profile] zillah975 2012-04-29 01:32 pm (UTC)(link)
Oh my god, that's hilarious and awesome. GO YOU, [community profile] sixwordstories!! *scampers to see this place*
ardath_rekha: (Default)

[personal profile] ardath_rekha 2012-04-29 02:07 pm (UTC)(link)
I went and checked out SWS -- because I played in it periodically back in the day -- and there's something odd you might want to check out. I was kinda surprised that even SWS would have hit that mark so quickly... it was big, but even back in its LJ heyday I hadn't thought it was that magnitude. It appears that the comment import may have run incorrectly and, at least in some posts, imported the same comments multiple times. You can see it pretty clearly in this post, where the same set of threads repeats three or four times as you scroll down the page.

A lot of the comment counts look pretty high, but that's the only one I checked before popping back in here to say something, because it was the first one I opened (one of my characters periodically played with the OP of that thread and I couldn't remember if that post was one of the times she did... it wasn't).
wytchcroft: art (infinite)

[personal profile] wytchcroft 2012-04-29 02:33 pm (UTC)(link)
*laugh* thanks as ever for the info...
and compared to the current problems with lj - i think i'll cope! :)
perlandria: Rainbow Fractal Mandlebrot (Rainbow Fractal Mandlebrot)

[personal profile] perlandria 2012-04-29 04:37 pm (UTC)(link)
Thank you for all your hard work! I love this place.
mlyn: (Default)

[personal profile] mlyn 2012-04-29 05:24 pm (UTC)(link)
As always, I deeply appreciate your transparency and broadly informative updates on these kinds of problems. Two thumbs up. :)
random_ooc: The words Tam I am on a purple background (Default)

[personal profile] random_ooc 2012-05-02 08:49 pm (UTC)(link)
Yes. DW is wonderful. The only problem that still frustrates me is the occasional import that says it's successful, but only imports one entry. And the lack of support when I ask for help on them. Compared to my list of complaints against LJ, that's nothing. lol

[personal profile] ladyofmarvel 2012-05-08 12:11 pm (UTC)(link)
I'm not quite sure how long the transfer is intending to take, but I'm still having 503 errors when trying to load and/or reply to any sixwordstories post. It's doable, but it's just a pain in my butt to load the page - wait for it to try to load then spit out an error at me, then reload. Then have to go through all that again when I try to reply.

[personal profile] twoeviltwins 2012-05-08 11:59 pm (UTC)(link)
Hey just a heads up, guys, but for a few days, DW hasn't been sending new message notifications to my Email. Didn't know if this was just me or a larger problem, but droppin' ya an FYI just in case. Thanks!

[personal profile] ladyofmarvel 2012-05-14 04:40 pm (UTC)(link)
To clarify on the 503 error I'm still getting when trying to use/answer a [community profile] sixwordstories post.

I'm using Chrome. If I clean my cache and try to open or 'reply to' a [community profile] sixwordstories post, it will load right away. However, trying to post the reply gives me the 503 error. And if I try to open another sws post after that, I get the same thing. Having to clean my cache before loading every post is a little ridiculous.

This is what pops up on the page for me:
Error 503 Service Unavailable

Service Unavailable

Guru Meditation:

XID: 2017400525

Varnish cache server

Any news on when that will stop?

Me again

[personal profile] ladyofmarvel 2012-06-14 08:58 pm (UTC)(link)
Look, a month to the day. Aww, it's our anniversary. Just popping in to see if you guys have an estimate on when the massive glitching will stop on the sws comm yet. No rush, and I'm not bitching, just wondering. I've been making due on others, but it'd be nice to have that one open to me without the issues anymore :)