Site search

[denise]

We noticed that the site search was having some difficulties yesterday, so mark switched the indexing to run every 12 hours instead of every 6 hours. (This means that search results will now be delayed up to 12 hours, not up to 6.)

Sometime in the next few weeks, depending on when we can schedule it for best results, we'll be taking search offline for about a day or so to do some upgrade and efficiency work on the machine that hosts it. (As mark called it, "spring cleaning".) We'll let you know when.

Comments

[par_avion]

Thanks for letting us know!

I <3 site search :)

[etfb]

I have a fair few blog entries, and if I search for something more than a couple of years old it generally times out. I usually just try again with a different sort order, or else use Google (adding "site:etfb.dreamwidth.org" to the query). I figured Search was a pretty low priority task, so I'm glad to see it's getting some love.

[denise]

It's not so much that it's low-priority as search is a really hard thing to do within the confines of the resources we have! (For best results, honestly we should probably have about the same hardware dedicated to search as we have dedicated to database servers and webservers.)

But I love the feature, and it's so useful, so we try our best. *g*

(I am reminded of the time that we-at-LJ hired a search expert specifically to add the feature. He spent nearly a year trying, and still couldn't do it in the confines of the budget we could muster.)

[terrierlee]

Not about the search but...

For some reason my DW is no longer set to crosspost to LJ. I checked everything and it should crosspost, but when I go back to Edit Entry, the crosspost to LJ is unchecked again.

[denise]

Generally this means that there was some sort of failure on the crosspost; check your inbox for notifications, and if you can't figure it out, open up a support request!

[terrierlee]

They started up again.

Thanks though :D

[reddragdiva]

A useful example is the search on Wikipedia. Basically, it sucked for years. Now it's not too bad, but it requires serious server muscle - as you say, comparable to an unclustered database server - and they're now looking for a dev to tweak Lucene (the only open source text search anywhere that's better than shit, and that barely) to local needs since the last one is doing something else.

[iguana]

Have you looked at Xapian? It's also free and open source, and since it's C++ (with various language bindings) rather than Java I imagine it takes up considerably less system resource. I've used Xapian personally and professionally and it's lovely to work with.

[denise]

If you were talking to us, we use Sphinx!

[iguana]

Python <3

Sphinx has a clever little JS search engine, indeed. From the source code it looks like it does Python server-side indexing and JS client-side searching. Clever, but not going to be very speedy.

[denise]

*nod* We traded off speed for ease of implementation and maintenance -- it was what Mark knew. When it gets to the point where the indexing is too long to run, we'll start looking into switching to a more low-level language, but for now it's working for us!