mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
Mark Smith ([staff profile] mark) wrote in [site community profile] dw_maintenance2012-04-28 05:22 pm

Site slowness expected shortly

Hi all,

Summary: some unexpected but required maintenance is going to cause the site to slow down by some amount for the next hour or two.

Some details: We have run into an issue with one of our larger communities that requires us to make a change to the way we store comments in memcache. This means that our caches are all going to invalidate here in a few minutes when I roll the code out, so the databases are going to get very busy until we repopulate the caches.

Those of you who have been on LJ for a few years might remember when ONTD ran into an issue when they hit 16,777,215 comments and LJ had to do the same thing. I helped LJ with the code to fix the problem, but for some reason, never committed it on Dreamwidth. Now we have hit the same issue and I feel silly for having not done that, but here we are, and hence this maintenance.

Please let me know if you see any major problems, and again, sorry for the inconvenience. This really isn't a great time to be doing site maintenance.

Update: Congratulations to [community profile] sixwordstories for being the first community or journal to hit the 16.7 million comment limit in the database. :)

At any rate, currently the community is being migrated to a database where I fixed up the column widths so they won't have this limit anymore. It will take some time to move that much data, though. In the meantime the community will be in read-only mode.

I'm sorry that we hit this -- I really thought we had made this fix on Dreamwidth after LJ ran into it. It turns out we didn't, though, hence this situation. Alas.
marahmarie: Sheep go to heaven, goats go to hell (Default)

[personal profile] marahmarie 2012-04-29 01:37 am (UTC)(link)
Has DW really hit 16,777,215 comments in just one of the communities? Way to go! This site must be getting waaaaaay more traffic than I thought was probable.

Uh, sorry Mark, couldn't resist; but I *am* curious if your earlier statement about 2^24 comments in that comm and the numerical figure that tied up LJ are the same - again?
Edited 2012-04-29 01:38 (UTC)