Search is recovering, but backfilling may take a few days

[mark]

Hi all,

The search service is back up and running but it is having to backfill all of history. This process isn't the fastest thing in the world so it may take some time to fully complete. I will keep an eye on it and see how things are going, but I wouldn't be surprised if it takes a few days to get back to having all of history.

I did some more digging into what happened and it turns out it was a run-of-the-mill hardware failure. We got a notification two weeks ago that the hardware had degraded and, unfortunately, I didn't process it fast enough before it finally kicked the bucket.

For a little more gory details: we run a pretty boring, basic search system that runs on a single machine to save money. Since search data can be recreated at any time based on our (very redundant and regularly backed up) main database, we treat our search hardware as semi-expendable.

Anyway, it's back up and running and you can start searching again, although you may not find what you expect until the reindexing process has completed.

Thanks for your patience, as always.

-Mark

Comments

[goodbyebird]

🙌🙌🙌

[vriddy]

Appreciate the explanations, glad it's back!

[luckyzukky]

thanks for your hard work and the updates!

[freyjaw]

Thank you!

[falena]

Thanks.

[vass]

Thanks for keeping us posted.

[subsequent]

Cheers for the updates! :)

[darth_eldritch]

Thank you for your hard work!

[goss]

Lovely, thank you! :)

Thank you for the update

[akatsija]

         *                  *
             __                *
          ,db'    *     *
         ,d8/       *        *    *
         888
         `db\       *     *
           `o`_                    **
      *               *   *    _      *
            *                 / )
         *    (\__/) *       ( (  *
       ,-.,-.,)    (.,-.,-.,-.) ).,-.,-.
      | @|  ={      }= | @|  / / | @|o |
     _j__j__j_)     `-------/ /__j__j__j_
     ________(               /___________
      |  | @| \              || o|O | @|
      |o |  |,'\       ,   ,'"|  |  |  |  hjw
     vV\|/vV|`-'\  ,---\   | \Vv\hjwVv\//v
                _) )    `. \ /
               (__/       ) )
                         (_/

--
А.Кац и Я

[paserbyp]

Thank you for explanation and question. It’s cheaper run search on AWS than on your own hardware?

[mark]

Running everything on actual hardware would be cheaper in pure dollar cost, but I'm willing to pay for the flexibility of cloud servers so that I don't have to deal with actually buying hardware and dealing with failures, remote hands, etc.

We actually started out by hosting in a data center and leasing hardware back when we first started... later, I did the math and doing 3-year commits on AWS wasn't that much more expensive, so we migrated.

[paserbyp]

Thank you!

[falkner]

Thank you for your hard work!

[runpunkrun]

Thanks for keeping us updated on this!

[muccamukk]

Thanks for the update! It'll be great to have the search back, and I appreciate your work on this.

[elaiel]

Thanks, for everything, DW is the best.

Strong Work

[degringolade]

I do agree with you with keeping you search on a single machine. Sound plan and keeps cost low and beer and pizza within reach.

I appreciate what you guys do. Dreamwidth is a pretty nice home.

John

[pronker]

applause

[thistleingrey]

Thank you!

[brickhousewench]

Nerdy question, are you using Solr or something else for search?

[mark]

We are using Sphinx actually. Our use case is pretty simple and this works well enough for us.

[brickhousewench]

Thanks for satisfying my curiosity!

[yourlibrarian]

Yay, thank you!

[pauamma]

Thanks for the update. While on that topic, any chance of a user interface knob to turn off stemming in searches, as I mentioned in https://www.dreamwidth.org/support/see_request?id=38865 and in the suggestion I made that may still be in the moderation queue?

[mark]

With the current search backend (Sphinx), to do this we'd have to maintain a separate search index that has stemming disabled and then let the searcher specify which index to use. This is possible of course, but I've never investigated how heavy maintaining a completely separate search index would be.

[pauamma]

Ah, probably not worth (unless and until you switch indexers) it if it's a hard backend setting. Thanks.

[mekare]

Thank you for the update and all your hard work!

Thanks a lot for your work!

[math_mommy]

I wonder if you have a policy against people using your platform for spreading misinformation.

Re: Thanks a lot for your work!

[mark]

If you would like to report something that may be against the terms of service, please let us know here: https://www.dreamwidth.org/support/submit

Our staff will take a look and let you know if we feel the content is against our terms of service.