WhitespaceTokenizerFactory 和 StandardTokenizerFactory 的区别 [英] Difference between WhitespaceTokenizerFactory and StandardTokenizerFactory

问题描述

我是 Solr 的新手.通过阅读 Solr 的 wiki，我不明白 WhitespaceTokenizerFactory 和 StandardTokenizerFactory 之间的区别.它们的真正区别是什么?

I am new to Solr. By reading Solr's wiki, I don't understand the differences between WhitespaceTokenizerFactory and StandardTokenizerFactory. What's their real difference?

推荐答案

它们的不同之处在于将分析的文本拆分为标记的方式.

They differ in how they split the analyzed text into tokens.

StandardTokenizer 基于以下内容(取自 lucene javadoc)执行此操作:

The StandardTokenizer does this based on the following (taken from lucene javadoc):

• 在标点字符处拆分单词，删除标点符号.但是，后面没有空格的点被认为是部分一个令牌.
• 在连字符处拆分单词，除非其中包含数字令牌，在这种情况下，整个令牌被解释为一个产品号且不拆分.
• 识别电子邮件地址和互联网主机名作为一个标记.

WhitespaceTokenizer 根据空白字符执行此操作:

The WhitespaceTokenizer does this based on whitespace characters:

WhitespaceTokenizer 是一种在空白处划分文本的标记器.相邻的非空白字符序列形成标记.

A WhitespaceTokenizer is a tokenizer that divides text at whitespace. Adjacent sequences of non-Whitespace characters form tokens.

您应该选择最适合您的应用程序的标记器.在任何情况下，您都必须使用相同的分析器/标记器进行索引和搜索！

You should pick the tokenizer that best fits your application. In any case you have to use the same analyzer/tokenizers for indexing and searching!

这篇关于WhitespaceTokenizerFactory 和 StandardTokenizerFactory 的区别的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！