Search changes

[mark]

Hi all,

I just rolled out some search changes. The two main things you should notice: search indexing is much quicker now (it updates with new content every 15 minutes) and also now comments (on paid accounts) are indexed.

Thank you for all of your patience with search. We outgrew our old trusty search machine and it's now retired. The new one is much larger, has faster disks, and should hold us in good standing for a while as we grow.

I'll be around all day and watching this post. Please let me know if you see any problems!

Comments

[alierak]

Looks like Sphinx can do this but we've left Cyrillic chars out of our charset_table. I guess we would need to add something like "U+410..U+42F->U+430..U+44F, U+430..U+44F" and then reindex everything? mark?

[mark]

We'd also want to look at using Russian stemming in this separate index. We'd have to figure out how to give the users a frontend where they can choose to search Russian vs English. (Or we could try to auto-detect it? Language auto-detection and then build a separate table?)

I'm happy to do something like this -- it'd be nice to better support our non-English users... I'm not sure what the right way to do it is...

I suppose I could do an easy test with the delta index. Let me away to do that...

[victor_sudakov]

Sphinx most certainly can do Cyrillic searches, at least sphinxsearch-devel-2.0.1b_1 on FreeBSD does that for me.