如何在Lucene 5.0中使用ngram标记器? [英] How to have ngram tokenizer in lucene 5.0?
本文介绍了如何在Lucene 5.0中使用ngram标记器?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我想为一个字符串生成ngram字符.以下是我使用的Lucene 4.1库.
I want to generate ngram characters for a string. Below is the Lucene 4.1 lib I used for it.
Reader reader = new StringReader(text);
NGramTokenizer gramTokenizer = new NGramTokenizer(reader, 3, 5); //catch contiguous sequence of 3, 4 and 5 characters
CharTermAttribute charTermAttribute = gramTokenizer.addAttribute(CharTermAttribute.class);
while (gramTokenizer.incrementToken()) {
String token = charTermAttribute.toString();
System.out.println(token);}
但是,我想使用Lucene 5.0.0来做到这一点. NGramTokenizer在Lucene 5.0.0中与以前的版本相比有很大变化,请参阅
However, I want to use Lucene 5.0.0 to do it. The NGramTokenizer changes a lot in Lucene 5.0.0 from the previous version, refer to http://lucene.apache.org/core/5_0_0/analyzers-common/index.html?org/apache/lucene/analysis/ngram/NGramTokenizer.html.
任何人都知道如何使用Lucene 5.0.0来执行ngram吗?
Anyone knows how to use Lucene 5.0.0 to do ngrams?
推荐答案
以下代码:
StringReader stringReader = new StringReader("abcd");
NGramTokenizer tokenizer = new NGramTokenizer(1, 2);
tokenizer.setReader(stringReader);
tokenizer.reset();
CharTermAttribute termAtt = tokenizer.getAttribute(CharTermAttribute.class);
while (tokenizer.incrementToken()) {
String token = termAtt.toString();
System.out.println(token);
}
将产生:
a
ab
b
bc
c
cd
d
这篇关于如何在Lucene 5.0中使用ngram标记器?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文