I guess is depends on the format you want your users to use when searching, to use asterisks for JP searches (infix method) or to use quotes around multiple characters that they want to be searched as a single word (for the ngram approach).

Thanks again for all your help.

THe Ngram feature treats each character as a white-space bounded word. But since there are not morphological analyzers that talk to Sphinx natively, you end up getting lots of characters that do not really match to any sensible string of Japanese vocablary.

The equivalent is searching for “dog” but Sphinx can only find “d” or “o” or “g”. This may have changed since early 2009, but I haven’t retested the Ngram feature.

On the other hand, the *infix* searching (basically a substring search) works very fast and matches pretty much anything as a string of characters.

Please let me know if you have a different experience? B)

What about the ngram characters? Are they not required for the jp charset? On http://www.sphinxsearch.com/wiki/doku.php?id=charset_tables they advise the use these for the CJK charsets.

See here: http://www.sphinxsearch.com/wiki/doku.php?id=charset_tables#cjk_ngram_characters

The differences are as follows:

1. Half-width & full-width kana characters are mapped as the same character, reducing confusion when users search using full-width text against a document with half-width text. For the uninitiated, half-width katakana characters are an essential part of the Japanese banking system, and other legacy systems, but read exactly the same way as their full-width cousins. Read this for a more colorful explanation http://www.ops.ietf.org/lists/idn/idn.2001/msg02452.html.

2. Glottal (eg. ka > ga) and aspirated (eg. ha > pa/ba) phonemes are treated as the same as non-glottal/aspirated. This is important because people often treat these differently in speech, and therefore tend to get them wrong when searching. I believe it’s better to match more widely, and then let them choose. Admittedly this could be omitted, and you can expand the search keywords programatically before sending it to Sphinx. The UTF range is clearly marked for your/my convenience further down the track!

3. This character map only includes the J from CJK. So it should index source documents faster, but of course will exclude any Korean or Chinese characters. It also excludes the most rare Japanese characters because these tend to be represented as kana these days (as a part of the general dumbing down of the Japanese writing system … thank God!)

This map was what “I” really needed for a project and at the time I could only find a documented map for Chinese text. This map will of course match all ASCII characters, which are an essential part of writing Japanese anyway (believe it or not!) There is more information at http://www.unicode.org if you are interested in digging deeper.

Thanks also for your words of encouragement.

DISCLAIMER: Apologies for any inaccuracies in this comment, I typed this off the top of my head.