如何使用 Lucene 获取频繁出现的短语 [英] How to get frequently occurring phrases with Lucene

问题描述

我想用 Lucene 获得一些经常出现的短语.我从 TXT 文件中获取了一些信息，并且由于没有短语信息而丢失了很多上下文，例如信息检索"被索引为两个单独的词.

I would like to get some frequently occurring phrases with Lucene. I am getting some information from TXT files, and I am losing a lot of context for not having information for phrases e.g. "information retrieval" is indexed as two separate words.

有什么方法可以得到这样的短语?我在互联网上找不到任何有用的东西，所有的建议、链接、提示，尤其是示例，都非常感谢！

What is the way to get the phrases like this? I can not find anything useful on internet, all the advices, links, hints especially examples are appreciated!

我只按标题和内容存储我的文档:

I store my documents just by title and content:

 Document doc = new Document();
 doc.add(new Field("name", f.getName(), Field.Store.YES, Field.Index.NOT_ANALYZED));
 doc.add(new Field("text", fReader, Field.TermVector.WITH_POSITIONS_OFFSETS));

因为对我来说最重要的是文件的内容.标题通常根本不具描述性(例如，我有许多 PDF 学术论文，其标题是代码或数字).

because for what I am doing the most important is the content of the file. Titles are too often not descriptive at all (e.g., I have many PDF academic papers whose titles are codes or numbers).

我迫切需要从文本内容中索引出现频率最高的短语，刚才我发现这种简单的词袋"方法效率不高.

I desperately need to index top occurring phrases from text contents, just now I see how much this simple "bag of words" approach is not efficient.

推荐答案

Julia，看来你要找的是 n-grams，特别是 Bigrams(也称为搭配).

Julia, It seems what you are looking for is n-grams, specifically Bigrams (also called collocations).

这里有一个 关于寻找搭配的章节 (PDF)，来自 Manning 和 Schutze 的 统计自然语言处理基础.

Here's a chapter about finding collocations (PDF) from Manning and Schutze's Foundations of Statistical Natural Language Processing.

为了使用 Lucene 做到这一点，我建议使用 Solr 和 SingleFilterFactory.有关详细信息，请参阅此讨论.

In order to do this with Lucene, I suggest using Solr with ShingleFilterFactory. Please see this discussion for details.

这篇关于如何使用 Lucene 获取频繁出现的短语的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！