解决Android SQLite全文搜索亚洲文本 [英] Work around of Android SQLite full-text search for Asian text

问题描述

我已阅读过很多文章，询问基于SQLite的全文搜索是否可以在Android中完成，并且所有答案都指出Android的内置SQLite不允许自定义标记器。默认的分词器会考虑用空格或其他符号分隔的单词，但亚洲词汇（如中文）需要特殊的分词器，但Android不允许添加自定义分词器。

I have read about many posts asking whether the SQLite based full-text search can be done in Android, and all the answers point out that the built-in SQLite of Android does not allow custom tokenizer. The default tokenizer considers the words separated by space or other signs, but Asian words (like Chinese) need its special tokenizer, but Android does not allow adding custom one.

我读过的帖子是多年前的。最近的Android版本有没有更新？我刚刚搜索并没有找到答案。

The posts I read were years ago. Is there any update in recent Android versions? I just searched and did not find an answer.

我正在考虑解决问题。在将元组插入到FTS3 / FTS4虚拟表中进行索引之前，我是否可以在每个词之间人为地添加空格，以便默认分词器可以将每个亚洲词视为英语单词？在执行查询时，查询字符串的作用相同，也添加了人造空间。

And I am thinking a work-around. Is it feasible that before I INSERT tuples into the FTS3/FTS4 virtual table for indexing, I artificially add spaces between each word, so that the default tokenizer can consider each Asian "word" like an English word? When doing the query, the query string does the same, that artificial spaces are also added.

我在Linux中尝试过，看起来像是有效。例如，如果我这样做，亚洲文本的全文搜索就可以了：

I tried in Linux, looks like it works. For example, if I do like this, full-text search is OK for Asian texts:

CREATE VIRTUAL TABLE mail USING fts3(subject, body);
INSERT INTO mail(docid, subject, body) VALUES(4, 'software feedback', '这 个 Bug 还 没 有 解 决');
SELECT * FROM mail WHERE body MATCH '没 有 解 决';  

但有一个疑问是，数据库文件是否会花费更多的存储空间，与空间的字符。它看起来像所谓的虚拟表不仅存储生成的索引，而且还存储原始文本。

But one doubt is that whether it would cost much more storage for the database file, as there are double of characters with the spaces. It looks like the so called "virtual table" not only stores the generated index, but also the original text.

推荐答案

对于API Level 21或更高版本，我测试并发现ICU标记器已经可用。

对于旧设备，我找到了解决方法另一个问题：
是否为启用FTS的ICU标记生成器构建的Android上的SQLite？

For older devices, I found a work-around solution in another question: Is SQLite on Android built with the ICU tokenizer enabled for FTS?

这篇关于解决Android SQLite全文搜索亚洲文本的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！