I copied your character table as well as the settings but cannot get any matches whatsoever from the japanese entries. The Latin (english, spanish) entries are fine and return matches, even with accents in spanish, but nothing in Japanese.

I also tried the configuration in the Sphinx wiki with ngram settings and still nothing. Any help you could give me would be greatly appreciated.

As an example, Our database will indeed find the following using a LIKE search:

…WHERE title LIKE ‘%レプリカントワークス%’;

But Sphinx will not, even from the CLI:

../bin/search -i bookIndex レプリカントワークス

That is a title copied from our database.

Thanks in advance.

http://sphinxsearch.com/docs/current.html#ref-indextool

Also, are you accessing the daemon through a library like Thinking Sphinx?
I use Thinking Sphinx in RoR, but I remember using the command line directly
once to debug things.

The CLI query tool included is called ‘search’ – funnily enough. See if you can return
anything from your indexes at the command line. Try both standard ASCII and CJK characters too.

http://sphinxsearch.com/docs/current.html#ref-search

Alternatively, send me some debug and I’ll see what I can do.

I have a question regarding sphinx index configuration (it’s actually nothing to do with Japanese).

My index file is too big (more than 4 GB) that the searchd throws a warning and it wouldn’t start. I looked for a solution and found out that I can split one big index into multiple indexes, but I don’t know how to do that…

If you happen to know how to write a conf file to build multiple indexes from a same source, could you show me the example?

Thanks very much!

Thanks!

I guess is depends on the format you want your users to use when searching, to use asterisks for JP searches (infix method) or to use quotes around multiple characters that they want to be searched as a single word (for the ngram approach).

Thanks again for all your help.

THe Ngram feature treats each character as a white-space bounded word. But since there are not morphological analyzers that talk to Sphinx natively, you end up getting lots of characters that do not really match to any sensible string of Japanese vocablary.

The equivalent is searching for “dog” but Sphinx can only find “d” or “o” or “g”. This may have changed since early 2009, but I haven’t retested the Ngram feature.

On the other hand, the *infix* searching (basically a substring search) works very fast and matches pretty much anything as a string of characters.

Please let me know if you have a different experience? B)

What about the ngram characters? Are they not required for the jp charset? On http://www.sphinxsearch.com/wiki/doku.php?id=charset_tables they advise the use these for the CJK charsets.

See here: http://www.sphinxsearch.com/wiki/doku.php?id=charset_tables#cjk_ngram_characters

The differences are as follows:

1. Half-width & full-width kana characters are mapped as the same character, reducing confusion when users search using full-width text against a document with half-width text. For the uninitiated, half-width katakana characters are an essential part of the Japanese banking system, and other legacy systems, but read exactly the same way as their full-width cousins. Read this for a more colorful explanation http://www.ops.ietf.org/lists/idn/idn.2001/msg02452.html.

2. Glottal (eg. ka > ga) and aspirated (eg. ha > pa/ba) phonemes are treated as the same as non-glottal/aspirated. This is important because people often treat these differently in speech, and therefore tend to get them wrong when searching. I believe it’s better to match more widely, and then let them choose. Admittedly this could be omitted, and you can expand the search keywords programatically before sending it to Sphinx. The UTF range is clearly marked for your/my convenience further down the track!

3. This character map only includes the J from CJK. So it should index source documents faster, but of course will exclude any Korean or Chinese characters. It also excludes the most rare Japanese characters because these tend to be represented as kana these days (as a part of the general dumbing down of the Japanese writing system … thank God!)

This map was what “I” really needed for a project and at the time I could only find a documented map for Chinese text. This map will of course match all ASCII characters, which are an essential part of writing Japanese anyway (believe it or not!) There is more information at http://www.unicode.org if you are interested in digging deeper.

Thanks also for your words of encouragement.

DISCLAIMER: Apologies for any inaccuracies in this comment, I typed this off the top of my head.