Search changes

[mark]

Hi all,

I just rolled out some search changes. The two main things you should notice: search indexing is much quicker now (it updates with new content every 15 minutes) and also now comments (on paid accounts) are indexed.

Thank you for all of your patience with search. We outgrew our old trusty search machine and it's now retired. The new one is much larger, has faster disks, and should hold us in good standing for a while as we grow.

I'll be around all day and watching this post. Please let me know if you see any problems!

Comments

[foxfirefey]

Holy moly that is hella faster and comments are coming up beautifully!

[mark]

A RAID-1 of two SSDs plus 24GB of RAM... the old machine was two separate SATA drives and 8GB. Yeah. I like the new one a lot. :)

[fu]

The new one is pretty cool. I shall hug it and love it and call it george.

SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND

[sepdet]

"The new one is much larger, has faster dicks, and should hold us in good standing for a while as we grow."

Whhaaaaaat? Oh. Right. DISKS.


Er, yes. Thank you! Go team!

Re: SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND

[mark]

Hahahaha, awesome. You're welcome! :)

Re: SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND

[denise]

I am about four years overdue for an eye exam + new glasses, and I make misreadings like that ALL THE TIME. I can't remember what it was I misread today, but the misreading was "chastity".

Re: SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND

[tameiki]

"The new one is much larger, has faster dicks, and should hold us in good standing for a while as we grow."

Whhaaaaaat? Oh. Right. DISKS.

Oh. OW! I think I hurt myself laughing. Best update EVAR! =D

Re: SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND

[rydra_wong]

Countdown until the first dw_maintenance porn ...

Or until someone starts a "Faster Dicks" ficathon.

(Unrelatedly, I have to admire your fine icon. Is that your cat confronting beets?)

Re: SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND

[azurelunatic]

I declare the "Faster Dicks" ficathon on-topic for beginningcocks.

Re: SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND

[sepdet]

It is, in fact, my cat confronting lumpy radishes and misshapen Danvers carrots, first fruits of the Garden of Suburbia.
and I totally didn't lure her into position with a wad of fresh catnip to stage the shot.

Also, huzzah for fic!

Re: SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND

[rydra_wong]

Yes, that does look a little like the wild-eyed catnip stare there, ready to pounce the moment one of those radishes makes a false move ..

[everlastingsoul]

This is a great change! I was pleasantly surprised to see new search results when I put in regular interests I search for. Thank you for all your efforts!

[jjhunter]

Hip hip hooray! Search is working gorgeous quick now; thank you so much, mark for getting it back up and running better than ever. :o)

[shehasathree]

Oh wow, that is awesome. :D

[cazzasaurus]

"also now comments (on paid accounts) are indexed"

As in we can now search for public comments made by a certain user?

[mark]

No, you can't search by author; it's all text based. If you happen to know that they always sign their comments by some unique words, you could perhaps search for that -- but it'd still be very hit or miss (and most people don't have a unique sign-off like that).

Also, comments are only indexed if they are on a paid account. Not if they were made BY a paid account -- but if they are ON a paid account. This is mostly designed for large communities that want to be able to search through comments, as a lot of the content is in the comments.

[azurelunatic]

Comment subjects too, or only comment bodies?

[fu]

Comment subjects, too, same with entries!

[azurelunatic]

Yay!

[owl]

Awesome!

[lannamichaels]

*\o/*

[victor_sudakov]

It still does not search Cyrillic text. What a shame.

[alierak]

Looks like Sphinx can do this but we've left Cyrillic chars out of our charset_table. I guess we would need to add something like "U+410..U+42F->U+430..U+44F, U+430..U+44F" and then reindex everything? mark?

[mark]

We'd also want to look at using Russian stemming in this separate index. We'd have to figure out how to give the users a frontend where they can choose to search Russian vs English. (Or we could try to auto-detect it? Language auto-detection and then build a separate table?)

I'm happy to do something like this -- it'd be nice to better support our non-English users... I'm not sure what the right way to do it is...

I suppose I could do an easy test with the delta index. Let me away to do that...

[victor_sudakov]

Sphinx most certainly can do Cyrillic searches, at least sphinxsearch-devel-2.0.1b_1 on FreeBSD does that for me.

[mark]

Hi,

I'm doing an experiment on the index of recent content only (last few hours). Can you try searching for some stuff and see if it works?

If it does, I will roll the change out to the global index and re-index the whole dataset.

--Mark

[victor_sudakov]

It seems to work now with one limitation: it searches only for exact matches, not word forms. I also tried searching with wildcards, it does not work either.

Even in this way, it would be a great improvement.

[mark]

The exact matches only is because of stemming. It looks like I can set two stemming libraries -- although it prefers the English one. If that doesn't match, then it will try the Russian one.

Can you do some searches now and see if it's better? It should match things like "testing" when you search for "test" (except, the Russian equivalent).

[victor_sudakov]

There is no difference, sorry to say. It still finds only exact matches for Cyrillic text.

[mark]

Can you give me a word that should be stemmed (i.e., a non-plural, or present tense, etc)? Then I can experiment and, ideally, have a quicker test turnaround. :-)

[victor_sudakov]

OK, take the initial form (Nominative case, singular) of the word общение (meaning communication) and try to search for its different forms like общения общению общением общении

This word is present here: http://victor-sudakov.dreamwidth.org/162749.html in the prepositional case.

[mark]

Thank you -- found the problem. It appears to work for me, can you test and confirm?

[victor_sudakov]

Hurrah, it works! Thank you Mark!

[kore]

Wow, nice! It's great to see DW improving -- I love it.

[saphirablue]

Um, I don't want my comments to be indexed and searchable. Any way to turn that off?

[mark]

It works the same as the "attempt to block outside search engine" settings -- we obey the preferences of the location of the comment. If you disable global search on your journal, comments on your journal (no matter who they're by) won't be publicly searchable.

The same goes for other accounts. If a community or journal disables global searching of their account, then content there (including comments) will not be publicly searchable. If they have chosen to let their account be indexed, though, then we will index the content on their accounts.

To manage your search related settings, please visit the Privacy tab of the Manage Account page.

[valsmith706]

Wow! A lot of people, though...

[sharpiefan]

Help!! I'm using Practicality, and have had no issues, have done nothing to the layout any time recently... and the sidebar is mostly off-screen on the left when I'm on my reading page. (It's fine in my journal though).

Have a screenshot:

The paler bit all the way to the left is my sidebar which should be sitting in that dark green blank space.

Here's a screenshot of what it ought to look like:

This has only happened to me this morning, and I know I've done nothing, so it's got to be someone else, somewhere. Help?

[denise]

it's almost certainly somebody on your reading page posting something with broken HTML! it's not happening for me, so it's likely something in a protected entry that you have access to. If you can figure out which entry it is, you might want to let the poster know. :)