mark: Photo of Mark's face, taken in standard office fluorescent. (me)Mark Smith ([staff profile] mark) wrote in [site community profile] dw_maintenance,
@ 2013-02-17 12:45 pm UTC
  • Previous Entry
  • Add to Memories
  • Tell someone about this!
  • Next Entry
Hi all,

I just rolled out some search changes. The two main things you should notice: search indexing is much quicker now (it updates with new content every 15 minutes) and also now comments (on paid accounts) are indexed.

Thank you for all of your patience with search. We outgrew our old trusty search machine and it's now retired. The new one is much larger, has faster disks, and should hold us in good standing for a while as we grow.

I'll be around all day and watching this post. Please let me know if you see any problems!

(39 comments) - (Post a new comment)
(Flat) (Top-level comments only)

foxfirefey: A seal making a happy face. (seal of approval)


[personal profile] foxfirefey
2013-02-17 09:50 pm UTC (link)
Holy moly that is hella faster and comments are coming up beautifully!

(Reply to this)  (Thread) 


mark: Photo of Mark's face, taken in standard office fluorescent. (me)


[staff profile] mark
2013-02-17 10:30 pm UTC (link)
A RAID-1 of two SSDs plus 24GB of RAM... the old machine was two separate SATA drives and 8GB. Yeah. I like the new one a lot. :)

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


fu: Close-up of Fu, bringing a scoop of water to her mouth (fu)


[staff profile] fu
2013-02-18 01:55 am UTC (link)
The new one is pretty cool. I shall hug it and love it and call it george.

(Reply to this)  (Thread from start)  (Parent) 


sepdet: Samhain worshipping the veggies. Oooommm. (Okay, yes, catnip was involved.) (garden)

SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND

Shifty Eyes
[personal profile] sepdet
2013-02-17 10:23 pm UTC (link)
"The new one is much larger, has faster dicks, and should hold us in good standing for a while as we grow."

Whhaaaaaat? Oh. Right. DISKS.


Er, yes. Thank you! Go team!

(Reply to this)  (Thread) 


mark: Photo of Mark's face, taken in standard office fluorescent. (me)

Re: SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND


[staff profile] mark
2013-02-17 10:30 pm UTC (link)
Hahahaha, awesome. You're welcome! :)

(Reply to this)  (Thread from start)  (Parent) 


denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (me, standing outside a broken phone booth)

Re: SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND


[staff profile] denise
2013-02-18 12:09 am UTC (link)
I am about four years overdue for an eye exam + new glasses, and I make misreadings like that ALL THE TIME. I can't remember what it was I misread today, but the misreading was "chastity".

(Reply to this)  (Thread from start)  (Parent) 


tameiki: (00 Teni with Tam name)

Re: SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND


[personal profile] tameiki
2013-02-18 02:24 am UTC (link)
"The new one is much larger, has faster dicks, and should hold us in good standing for a while as we grow."

Whhaaaaaat? Oh. Right. DISKS.

Oh. OW! I think I hurt myself laughing. Best update EVAR! =D

(Reply to this)  (Thread from start)  (Parent) 


rydra_wong: dreamsheep with spork and "SheepSpork" logo; no, it wouldn't make any more sense if you saw it (dreamwidth -- sheepspork)

Re: SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND


[personal profile] rydra_wong
2013-02-18 08:25 am UTC (link)
Countdown until the first [site community profile] dw_maintenance porn ...

Or until someone starts a "Faster Dicks" ficathon.

(Unrelatedly, I have to admire your fine icon. Is that your cat confronting beets?)

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


azurelunatic: cameo-like portrait of <user name="azurelunatic"> in short blue hair. (_support, cameo)

Re: SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND


[personal profile] azurelunatic
2013-02-18 08:28 am UTC (link)
I declare the "Faster Dicks" ficathon on-topic for [community profile] beginningcocks.

(Reply to this)  (Thread from start)  (Parent) 


sepdet: Samhain worshipping the veggies. Oooommm. (Okay, yes, catnip was involved.) (garden)

Re: SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND


[personal profile] sepdet
2013-02-18 09:53 am UTC (link)
It is, in fact, my cat confronting lumpy radishes and misshapen Danvers carrots, first fruits of the Garden of Suburbia.
and I totally didn't lure her into position with a wad of fresh catnip to stage the shot.

Also, huzzah for fic!

Last edited 2013-02-18 09:55 am UTC

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


rydra_wong: dreamsheep with spork and "SheepSpork" logo; no, it wouldn't make any more sense if you saw it (dreamwidth -- sheepspork)

Re: SOMETIMES I REALLY LOVE HAVING 20-40 VISION AHD A PRURIENT MIND


[personal profile] rydra_wong
2013-02-18 10:21 am UTC (link)
Yes, that does look a little like the wild-eyed catnip stare there, ready to pounce the moment one of those radishes makes a false move ..

(Reply to this)  (Thread from start)  (Parent) 


everlastingsoul: (Suikoden III - Young Silverberg brothers)


[personal profile] everlastingsoul
2013-02-17 10:42 pm UTC (link)
This is a great change! I was pleasantly surprised to see new search results when I put in regular interests I search for. Thank you for all your efforts!

(Reply to this) 


jjhunter: Drawing of human JJ in ink tinted with blue watercolor; woman wearing glasses with arched eyebrows (JJ inked)


[personal profile] jjhunter
2013-02-17 11:13 pm UTC (link)
Hip hip hooray! Search is working gorgeous quick now; thank you so much, [staff profile] mark for getting it back up and running better than ever. :o)

(Reply to this) 


shehasathree: (butterflies)


[personal profile] shehasathree
2013-02-17 11:48 pm UTC (link)
Oh wow, that is awesome. :D

(Reply to this) 


cazzasaurus: (PJA - BORING)


[personal profile] cazzasaurus
2013-02-18 12:47 am UTC (link)
"also now comments (on paid accounts) are indexed"

As in we can now search for public comments made by a certain user?

(Reply to this)  (Thread) 


mark: Photo of Mark's face, taken in standard office fluorescent. (me)


[staff profile] mark
2013-02-18 01:12 am UTC (link)
No, you can't search by author; it's all text based. If you happen to know that they always sign their comments by some unique words, you could perhaps search for that -- but it'd still be very hit or miss (and most people don't have a unique sign-off like that).

Also, comments are only indexed if they are on a paid account. Not if they were made BY a paid account -- but if they are ON a paid account. This is mostly designed for large communities that want to be able to search through comments, as a lot of the content is in the comments.

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


azurelunatic: cameo-like portrait of <user name="azurelunatic"> in short blue hair. (_support, cameo)


[personal profile] azurelunatic
2013-02-18 01:18 am UTC (link)
Comment subjects too, or only comment bodies?

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


fu: Close-up of Fu, bringing a scoop of water to her mouth (fu)


[staff profile] fu
2013-02-18 01:52 am UTC (link)
Comment subjects, too, same with entries!

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


azurelunatic: cameo-like portrait of <user name="azurelunatic"> in short blue hair. (_support, cameo)


[personal profile] azurelunatic
2013-02-18 02:59 am UTC (link)
Yay!

(Reply to this)  (Thread from start)  (Parent) 


owl: Jesse Eisenberg as Mark Zuckerberg (mzuck)


[personal profile] owl
2013-02-18 11:07 am UTC (link)
Awesome!

(Reply to this)  (Thread from start)  (Parent) 


lannamichaels: Astronaut Dale Gardner holds up For Sale sign after EVA. (astronomy, earth for sale, for sale)


[personal profile] lannamichaels
2013-02-18 01:50 am UTC (link)
*\o/*

(Reply to this) 


victor_sudakov: (pic#929745)


[personal profile] victor_sudakov
2013-02-18 02:27 pm UTC (link)
It still does not search Cyrillic text. What a shame.

(Reply to this)  (Thread) 


alierak: (default)


[personal profile] alierak
2013-02-18 05:20 pm UTC (link)
Looks like Sphinx can do this but we've left Cyrillic chars out of our charset_table. I guess we would need to add something like "U+410..U+42F->U+430..U+44F, U+430..U+44F" and then reindex everything? [staff profile] mark?

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


mark: Photo of Mark's face, taken in standard office fluorescent. (me)


[staff profile] mark
2013-02-18 07:42 pm UTC (link)
We'd also want to look at using Russian stemming in this separate index. We'd have to figure out how to give the users a frontend where they can choose to search Russian vs English. (Or we could try to auto-detect it? Language auto-detection and then build a separate table?)

I'm happy to do something like this -- it'd be nice to better support our non-English users... I'm not sure what the right way to do it is...

I suppose I could do an easy test with the delta index. Let me away to do that...

(Reply to this)  (Thread from start)  (Parent) 


victor_sudakov: (pic#929745)


[personal profile] victor_sudakov
2013-02-19 03:16 am UTC (link)
Sphinx most certainly can do Cyrillic searches, at least sphinxsearch-devel-2.0.1b_1 on FreeBSD does that for me.

(Reply to this)  (Thread from start)  (Parent) 


mark: Photo of Mark's face, taken in standard office fluorescent. (me)


[staff profile] mark
2013-02-18 07:49 pm UTC (link)
Hi,

I'm doing an experiment on the index of recent content only (last few hours). Can you try searching for some stuff and see if it works?

If it does, I will roll the change out to the global index and re-index the whole dataset.

--Mark

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


victor_sudakov: (pic#929745)


[personal profile] victor_sudakov
2013-02-19 04:22 am UTC (link)
It seems to work now with one limitation: it searches only for exact matches, not word forms. I also tried searching with wildcards, it does not work either.

Even in this way, it would be a great improvement.

Last edited 2013-02-19 04:23 am UTC

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


mark: Photo of Mark's face, taken in standard office fluorescent. (me)


[staff profile] mark
2013-02-19 04:34 am UTC (link)
The exact matches only is because of stemming. It looks like I can set two stemming libraries -- although it prefers the English one. If that doesn't match, then it will try the Russian one.

Can you do some searches now and see if it's better? It should match things like "testing" when you search for "test" (except, the Russian equivalent).

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


victor_sudakov: (pic#929745)


[personal profile] victor_sudakov
2013-02-19 06:21 am UTC (link)
There is no difference, sorry to say. It still finds only exact matches for Cyrillic text.

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


mark: Photo of Mark's face, taken in standard office fluorescent. (me)


[staff profile] mark
2013-02-19 06:26 am UTC (link)
Can you give me a word that should be stemmed (i.e., a non-plural, or present tense, etc)? Then I can experiment and, ideally, have a quicker test turnaround. :-)

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


victor_sudakov: (pic#929745)


[personal profile] victor_sudakov
2013-02-19 06:40 am UTC (link)
OK, take the initial form (Nominative case, singular) of the word общение (meaning communication) and try to search for its different forms like общения общению общением общении

This word is present here: http://victor-sudakov.dreamwidth.org/162749.html in the prepositional case.

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


mark: Photo of Mark's face, taken in standard office fluorescent. (me)


[staff profile] mark
2013-02-19 08:05 am UTC (link)
Thank you -- found the problem. It appears to work for me, can you test and confirm?

(Reply to this)  (Thread from start)  (Parent)  (Thread) 


victor_sudakov: (pic#929745)


[personal profile] victor_sudakov
2013-02-19 05:06 pm UTC (link)
Hurrah, it works! Thank you Mark!

(Reply to this)  (Thread from start)  (Parent) 


kore: (Barbara Cooney - Persephone)


[personal profile] kore
2013-02-18 03:38 pm UTC (link)
Wow, nice! It's great to see DW improving -- I love it.

(Reply to this) 


saphirablue: (Moon)


[personal profile] saphirablue
2013-02-18 07:25 pm UTC (link)
Um, I don't want my comments to be indexed and searchable. Any way to turn that off?

(Reply to this)  (Thread) 


mark: Photo of Mark's face, taken in standard office fluorescent. (me)


[staff profile] mark
2013-02-18 07:41 pm UTC (link)
It works the same as the "attempt to block outside search engine" settings -- we obey the preferences of the location of the comment. If you disable global search on your journal, comments on your journal (no matter who they're by) won't be publicly searchable.

The same goes for other accounts. If a community or journal disables global searching of their account, then content there (including comments) will not be publicly searchable. If they have chosen to let their account be indexed, though, then we will index the content on their accounts.

To manage your search related settings, please visit the Privacy tab of the Manage Account page.

(Reply to this)  (Thread from start)  (Parent) 


valsmith706: (pic#5725697)


[personal profile] valsmith706
2013-02-21 12:20 pm UTC (link)
Wow! A lot of people, though...

(Reply to this) 


sharpiefan: Picture of a boat and soldiers (Navy and Marines)


[personal profile] sharpiefan
2013-02-23 09:59 am UTC (link)
Help!! I'm using Practicality, and have had no issues, have done nothing to the layout any time recently... and the sidebar is mostly off-screen on the left when I'm on my reading page. (It's fine in my journal though).

Have a screenshot: photo DWborked_zps663b5db5.jpg

The paler bit all the way to the left is my sidebar which should be sitting in that dark green blank space.

Here's a screenshot of what it ought to look like: photo DWfine_zps81378546.jpg

This has only happened to me this morning, and I know I've done nothing, so it's got to be someone else, somewhere. Help?

(Reply to this)  (Thread) 


denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (me, standing outside a broken phone booth)


[staff profile] denise
2013-02-23 08:10 pm UTC (link)
it's almost certainly somebody on your reading page posting something with broken HTML! it's not happening for me, so it's likely something in a protected entry that you have access to. If you can figure out which entry it is, you might want to let the poster know. :)

(Reply to this)  (Thread from start)  (Parent) 


(39 comments) - (Post a new comment)
(Flat) (Top-level comments only)