Search is recovering, but backfilling may take a few days

[mark]

Hi all,

The search service is back up and running but it is having to backfill all of history. This process isn't the fastest thing in the world so it may take some time to fully complete. I will keep an eye on it and see how things are going, but I wouldn't be surprised if it takes a few days to get back to having all of history.

I did some more digging into what happened and it turns out it was a run-of-the-mill hardware failure. We got a notification two weeks ago that the hardware had degraded and, unfortunately, I didn't process it fast enough before it finally kicked the bucket.

For a little more gory details: we run a pretty boring, basic search system that runs on a single machine to save money. Since search data can be recreated at any time based on our (very redundant and regularly backed up) main database, we treat our search hardware as semi-expendable.

Anyway, it's back up and running and you can start searching again, although you may not find what you expect until the reindexing process has completed.

Thanks for your patience, as always.

-Mark

Comments

[paserbyp]

Thank you for explanation and question. It’s cheaper run search on AWS than on your own hardware?

[mark]

Running everything on actual hardware would be cheaper in pure dollar cost, but I'm willing to pay for the flexibility of cloud servers so that I don't have to deal with actually buying hardware and dealing with failures, remote hands, etc.

We actually started out by hosting in a data center and leasing hardware back when we first started... later, I did the math and doing 3-year commits on AWS wasn't that much more expensive, so we migrated.

[paserbyp]

Thank you!