denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
Denise ([staff profile] denise) wrote in [site community profile] dw_maintenance2012-04-23 11:16 pm

Search indexing temporarily paused; search will be down sometime in the next day

We've got a disk that looks like it's about to fail in the machine we use to power site and journal search. We've turned off search indexing temporarily to reduce the number of writes to the disk (which will hopefully let us baby it along for a little while longer), and will be taking search entirely offline for a four-hour maintenance window with our hosting provider as soon as we can schedule a convenient time for the replacement -- either later this evening, or early tomorrow morning.

Indexing being disabled means that no new posts will show up in site search until after the replacement, and when the machine is entirely offline, visiting site or journal search will return a message saying that the feature is temporarily offline. Once the disk is replaced, it may take a day or two for the indexing to catch up with the stuff that was posted while indexing was off; we've been hovering around the point where it takes longer to run the script than the frequency with which we run the script, so it may take us a little while to catch up.

The data on the disk is completely backed up (and in a worst case scenario could be re-indexed anyway), so there won't be any risk. The only thing you should notice will be no new posts appearing in the search results until the replacement, and then a period of downtime while the replacement is performed.

(EDIT, 24 April 5AM EDT: The disk has now been replaced, the data is copied back over, and the indexing is running again and catching up on what was missed. And, for bonus points, [staff profile] mark found a place to optimize a bunch of database queries.)
kore: (Dreamwidth on the xkcd map)

[personal profile] kore 2012-04-24 07:11 am (UTC)(link)
I am continually impressed with just how well you guys communicate with your users and keep us informed. Thank you for being on top of it.

Yes, exactly.