WhitespaceTokenizerFactory 和 StandardTokenizerFactory 的区别 [英] Difference between WhitespaceTokenizerFactory and StandardTokenizerFactory
问题描述
我是 Solr 的新手.通过阅读 Solr 的 wiki,我不明白 WhitespaceTokenizerFactory 和 StandardTokenizerFactory 之间的区别.它们的真正区别是什么?
I am new to Solr. By reading Solr's wiki, I don't understand the differences between WhitespaceTokenizerFactory and StandardTokenizerFactory. What's their real difference?
推荐答案
它们的不同之处在于将分析的文本拆分为标记的方式.
They differ in how they split the analyzed text into tokens.
StandardTokenizer 基于以下内容(取自 lucene javadoc)执行此操作:
The StandardTokenizer does this based on the following (taken from lucene javadoc):
- 在标点字符处拆分单词,删除标点符号.但是,后面没有空格的点被认为是部分一个令牌.
- 在连字符处拆分单词,除非其中包含数字令牌,在这种情况下,整个令牌被解释为一个产品号且不拆分.
- 识别电子邮件地址和互联网主机名作为一个标记.
WhitespaceTokenizer 根据空白字符执行此操作:
The WhitespaceTokenizer does this based on whitespace characters:
WhitespaceTokenizer 是一种在空白处划分文本的标记器.相邻的非空白字符序列形成标记.
A WhitespaceTokenizer is a tokenizer that divides text at whitespace. Adjacent sequences of non-Whitespace characters form tokens.
您应该选择最适合您的应用程序的标记器.在任何情况下,您都必须使用相同的分析器/标记器进行索引和搜索!
You should pick the tokenizer that best fits your application. In any case you have to use the same analyzer/tokenizers for indexing and searching!
这篇关于WhitespaceTokenizerFactory 和 StandardTokenizerFactory 的区别的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!