Search changes

[mark]

Hi all,

I just rolled out some search changes. The two main things you should notice: search indexing is much quicker now (it updates with new content every 15 minutes) and also now comments (on paid accounts) are indexed.

Thank you for all of your patience with search. We outgrew our old trusty search machine and it's now retired. The new one is much larger, has faster disks, and should hold us in good standing for a while as we grow.

I'll be around all day and watching this post. Please let me know if you see any problems!

Comments

[victor_sudakov]

It still does not search Cyrillic text. What a shame.

[alierak]

Looks like Sphinx can do this but we've left Cyrillic chars out of our charset_table. I guess we would need to add something like "U+410..U+42F->U+430..U+44F, U+430..U+44F" and then reindex everything? mark?

[mark]

We'd also want to look at using Russian stemming in this separate index. We'd have to figure out how to give the users a frontend where they can choose to search Russian vs English. (Or we could try to auto-detect it? Language auto-detection and then build a separate table?)

I'm happy to do something like this -- it'd be nice to better support our non-English users... I'm not sure what the right way to do it is...

I suppose I could do an easy test with the delta index. Let me away to do that...

[victor_sudakov]

Sphinx most certainly can do Cyrillic searches, at least sphinxsearch-devel-2.0.1b_1 on FreeBSD does that for me.

[mark]

Hi,

I'm doing an experiment on the index of recent content only (last few hours). Can you try searching for some stuff and see if it works?

If it does, I will roll the change out to the global index and re-index the whole dataset.

--Mark

[victor_sudakov]

It seems to work now with one limitation: it searches only for exact matches, not word forms. I also tried searching with wildcards, it does not work either.

Even in this way, it would be a great improvement.

[mark]

The exact matches only is because of stemming. It looks like I can set two stemming libraries -- although it prefers the English one. If that doesn't match, then it will try the Russian one.

Can you do some searches now and see if it's better? It should match things like "testing" when you search for "test" (except, the Russian equivalent).

[victor_sudakov]

There is no difference, sorry to say. It still finds only exact matches for Cyrillic text.

[mark]

Can you give me a word that should be stemmed (i.e., a non-plural, or present tense, etc)? Then I can experiment and, ideally, have a quicker test turnaround. :-)

[victor_sudakov]

OK, take the initial form (Nominative case, singular) of the word общение (meaning communication) and try to search for its different forms like общения общению общением общении

This word is present here: http://victor-sudakov.dreamwidth.org/162749.html in the prepositional case.

[mark]

Thank you -- found the problem. It appears to work for me, can you test and confirm?

[victor_sudakov]

Hurrah, it works! Thank you Mark!