Site slowness expected shortly

[mark]

Hi all,

Summary: some unexpected but required maintenance is going to cause the site to slow down by some amount for the next hour or two.

Some details: We have run into an issue with one of our larger communities that requires us to make a change to the way we store comments in memcache. This means that our caches are all going to invalidate here in a few minutes when I roll the code out, so the databases are going to get very busy until we repopulate the caches.

Those of you who have been on LJ for a few years might remember when ONTD ran into an issue when they hit 16,777,215 comments and LJ had to do the same thing. I helped LJ with the code to fix the problem, but for some reason, never committed it on Dreamwidth. Now we have hit the same issue and I feel silly for having not done that, but here we are, and hence this maintenance.

Please let me know if you see any major problems, and again, sorry for the inconvenience. This really isn't a great time to be doing site maintenance.

Update: Congratulations to sixwordstories for being the first community or journal to hit the 16.7 million comment limit in the database. :)

At any rate, currently the community is being migrated to a database where I fixed up the column widths so they won't have this limit anymore. It will take some time to move that much data, though. In the meantime the community will be in read-only mode.

I'm sorry that we hit this -- I really thought we had made this fix on Dreamwidth after LJ ran into it. It turns out we didn't, though, hence this situation. Alas.

Comments

[subluxate]

It is what it is, regarding when you have to roll it out. Thank you for keeping us updated.

[birke]

No prob.

[0jack]

Oo, I just want to know which community. *grins* *is bored, wants to read*

Good luck with this! :)

[thene]

I figured maybe cfud but that's only at about twelve million comments...?

[antilove]

Thanks for the heads up.

[sophie]

The fact that a community on Dreamwidth is having the same issue as ONTD had makes me feel like little DW is growing up!

[edit: Of course, DW isn't so little now anyway. But still...]

[marahmarie]

OK, so correct if I'm wrong....imported comments would count toward this figure - and that comm likely has tons of imported comments...this *does* make a difference to me, because I'm trying to figure out which DW comm could possibly get that popular that fast (when it seems DW hasn't even been around that long or well-known long enough for that to all happen right here). Just curious...

[helens78]

Hee! Thanks for the heads up; yay for getting the fix committed, and it's always so nice to be informed that it doesn't seem like a terribly inconvenient time at all. ^_^ (I think this is maybe a geek mindset thing; if we know WHY!! something is happening, it's so much easier to deal with it, no matter when! Or maybe that's just me.)

[ilyena_sylph]

Not just you!

[grackle]

I feel the same way.

[ariestess]

No worries. I'd rather know ahead of time and be able to plan for it where applicable.

[marahmarie]

Has DW really hit 16,777,215 comments in just one of the communities? Way to go! This site must be getting waaaaaay more traffic than I thought was probable.

Uh, sorry Mark, couldn't resist; but I *am* curious if your earlier statement about 2^24 comments in that comm and the numerical figure that tied up LJ are the same - again?

[rosalynian]

Your openness with the userbase and letting us know what's up each step of the way is just the loveliest thing, thank you.

♥

It's so nice just being allowed some looks behind-the-scenes of what you're doing; it really helps the site feel friendly and accessible. I'm so glad I and so many others migrated here!

[sharpiefan]

This. Oh, absolutely THIS.

I love this place! :D

[heartequals]

Round of applause for sixwordstories! That's awesome!

[jeweledeyes]

Yaaay Calcifer! ♥ /random

[jeweledeyes]

If you have to have a problem, this seems like a good one to have, lol. Go DW and sixwordstories!

[zillah975]

Oh my god, that's hilarious and awesome. GO YOU, sixwordstories!! *scampers to see this place*

[ardath_rekha]

I went and checked out SWS -- because I played in it periodically back in the day -- and there's something odd you might want to check out. I was kinda surprised that even SWS would have hit that mark so quickly... it was big, but even back in its LJ heyday I hadn't thought it was that magnitude. It appears that the comment import may have run incorrectly and, at least in some posts, imported the same comments multiple times. You can see it pretty clearly in this post, where the same set of threads repeats three or four times as you scroll down the page.

A lot of the comment counts look pretty high, but that's the only one I checked before popping back in here to say something, because it was the first one I opened (one of my characters periodically played with the OP of that thread and I couldn't remember if that post was one of the times she did... it wasn't).

[mark]

Yup, that is what happened. I'm working with the mods to remove the duplicates, but it won't fix this particular problem -- the "counter" has already passed the 16.7 million limit we had. Even after removing the duplicates, those IDs won't ever get reused, so we had to fix it anyway.

(And to be clear, when the dupes are removed the profile page will drop back to the number it's supposed to have. The count will be accurate, it's just that the database will have empty spots that it can never re-use.)

[wytchcroft]

*laugh* thanks as ever for the info...
and compared to the current problems with lj - i think i'll cope! :)

[perlandria]

Thank you for all your hard work! I love this place.

[mlyn]

As always, I deeply appreciate your transparency and broadly informative updates on these kinds of problems. Two thumbs up. :)

[random_ooc]

Yes. DW is wonderful. The only problem that still frustrates me is the occasional import that says it's successful, but only imports one entry. And the lack of support when I ask for help on them. Compared to my list of complaints against LJ, that's nothing. lol

[random_ooc]

Also, doesn't it say something good about DW that it's a creative community that gets the most comments, instead of a slagging gossip community?

[ladyofmarvel]

I'm not quite sure how long the transfer is intending to take, but I'm still having 503 errors when trying to load and/or reply to any sixwordstories post. It's doable, but it's just a pain in my butt to load the page - wait for it to try to load then spit out an error at me, then reload. Then have to go through all that again when I try to reply.

[twoeviltwins]

Hey just a heads up, guys, but for a few days, DW hasn't been sending new message notifications to my Email. Didn't know if this was just me or a larger problem, but droppin' ya an FYI just in case. Thanks!

[ladyofmarvel]

I've started having a loss of notifs too. I don't know why, and it's not all my accounts. Mainly this one. But still.

:|

[ladyofmarvel]

To clarify on the 503 error I'm still getting when trying to use/answer a sixwordstories post.

I'm using Chrome. If I clean my cache and try to open or 'reply to' a sixwordstories post, it will load right away. However, trying to post the reply gives me the 503 error. And if I try to open another sws post after that, I get the same thing. Having to clean my cache before loading every post is a little ridiculous.

This is what pops up on the page for me:
Error 503 Service Unavailable

Service Unavailable

Guru Meditation:

XID: 2017400525

Varnish cache server

Any news on when that will stop?

[denise]

It's not just you, and it doesn't have anything to do with clearing your browser cache -- after anyone comments, on any entry, at least the first refresh will get a 503 error. (Long story short, it's because the things that get updated after someone comments in the community are choking on the size of the community.)

We'll get it fixed as soon as we can!

Me again

[ladyofmarvel]

Look, a month to the day. Aww, it's our anniversary. Just popping in to see if you guys have an estimate on when the massive glitching will stop on the sws comm yet. No rush, and I'm not bitching, just wondering. I've been making due on others, but it'd be nice to have that one open to me without the issues anymore :)

Re: Me again

[mark]

Thanks for the reminder. I've just committed a fix for this, but it's not live yet. I'm hoping to do a code push here in a few hours though, so it should be resolved then.