Java Lucene NGramTokenizer [英] Java Lucene NGramTokenizer

问题描述

我正在尝试将字符串标记为ngram.奇怪的是，在

I am trying tokenize strings into ngrams. Strangely in the documentation for the NGramTokenizer I do not see a method that will return the individual ngrams that were tokenized. In fact I only see two methods in the NGramTokenizer class that return String Objects.

这是我拥有的代码:

Reader reader = new StringReader("This is a test string");
NGramTokenizer gramTokenizer = new NGramTokenizer(reader, 1, 3);

1. 被标记的ngram在哪里?
2. 如何获取字符串/单词的输出?

我希望输出为:这是一个测试字符串，这是一个测试字符串，这是一个测试字符串.

I want my output to be like: This, is, a, test, string, This is, is a, a test, test string, This is a, is a test, a test string.

推荐答案

我不认为您会找到想要查找返回String的方法的内容.您需要处理属性 s.

I don't think you'll find what you're looking for trying to find methods returning String. You'll need to deal with Attributes.

应该像这样工作:

Reader reader = new StringReader("This is a test string");
NGramTokenizer gramTokenizer = new NGramTokenizer(reader, 1, 3);
CharTermAttribute charTermAttribute = gramTokenizer.addAttribute(CharTermAttribute.class);
gramTokenizer.reset();

while (gramTokenizer.incrementToken()) {
    String token = charTermAttribute.toString();
    //Do something
}
gramTokenizer.end();
gramTokenizer.close();

但是，如果此后需要重用，请确保将Tokenizer重置().

Be sure to reset() the Tokenizer it if it needs to be reused after that, though.

根据注释对单词而不是字符进行分组:

Tokenizing grouping of words, rather than chars, per comments:

Reader reader = new StringReader("This is a test string");
TokenStream tokenizer = new StandardTokenizer(Version.LUCENE_36, reader);
tokenizer = new ShingleFilter(tokenizer, 1, 3);
CharTermAttribute charTermAttribute = tokenizer.addAttribute(CharTermAttribute.class);

while (tokenizer.incrementToken()) {
    String token = charTermAttribute.toString();
    //Do something
}

这篇关于Java Lucene NGramTokenizer的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！