如何从 Lucene TokenStream 获取令牌? [英] How to get a Token from a Lucene TokenStream?

问题描述

我正在尝试使用 Apache Lucene 进行标记化，但我对从 TokenStream 获取令牌的过程感到困惑.

I'm trying to use Apache Lucene for tokenizing, and I am baffled at the process to obtain Tokens from a TokenStream.

最糟糕的是，我正在查看解决我的问题的 JavaDocs 中的评论.

The worst part is that I'm looking at the comments in the JavaDocs that address my question.

http://lucene.apache.org/java/3_0_1/api/core/org/apache/lucene/analysis/TokenStream.html#incrementToken%28%29

不知何故，应该使用AttributeSource，而不是Tokens.我完全不知所措.

Somehow, an AttributeSource is supposed to be used, rather than Tokens. I'm totally at a loss.

谁能解释一下如何从 TokenStream 中获取类似令牌的信息?

Can anyone explain how to get token-like information from a TokenStream?

推荐答案

是的，这有点令人费解(与好的方法相比)，但应该这样做:

Yeah, it's a little convoluted (compared to the good ol' way), but this should do it:

TokenStream tokenStream = analyzer.tokenStream(fieldName, reader);
OffsetAttribute offsetAttribute = tokenStream.getAttribute(OffsetAttribute.class);
TermAttribute termAttribute = tokenStream.getAttribute(TermAttribute.class);

while (tokenStream.incrementToken()) {
    int startOffset = offsetAttribute.startOffset();
    int endOffset = offsetAttribute.endOffset();
    String term = termAttribute.term();
}

新方式

根据 Donotello 的说法，TermAttribute 已被弃用，取而代之的是 CharTermAttribute.根据 jpountz(和 Lucene 的文档)，addAttribute 比 getAttribute 更可取.

The new way

According to Donotello, TermAttribute has been deprecated in favor of CharTermAttribute. According to jpountz (and Lucene's documentation), addAttribute is more desirable than getAttribute.

TokenStream tokenStream = analyzer.tokenStream(fieldName, reader);
OffsetAttribute offsetAttribute = tokenStream.addAttribute(OffsetAttribute.class);
CharTermAttribute charTermAttribute = tokenStream.addAttribute(CharTermAttribute.class);

tokenStream.reset();
while (tokenStream.incrementToken()) {
    int startOffset = offsetAttribute.startOffset();
    int endOffset = offsetAttribute.endOffset();
    String term = charTermAttribute.toString();
}

这篇关于如何从 Lucene TokenStream 获取令牌?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！