用 lucene 提取 tf-idf 向量 [英] Extract tf-idf vectors with lucene

问题描述

我已经使用 lucene 索引了一组文档.我还为每个文档内容存储了 DocumentTermVector.我写了一个程序，得到了每个文档的词频向量，但是如何获取每个文档的 tf-idf 向量呢?

I have indexed a set of documents using lucene. I also have stored DocumentTermVector for each document content. I wrote a program and got the term frequency vector for each document, but how can I get tf-idf vector of each document?

这是我在每个文档中输出词频的代码:

Here is my code that outputs term frequencies in each document:

Directory dir = FSDirectory.open(new File(indexDir));
    IndexReader ir = IndexReader.open(dir);
    for (int docNum=0; docNum<ir.numDocs(); docNum++) {
        System.out.println(ir.document(docNum).getField("filename").stringValue());
        TermFreqVector tfv = ir.getTermFreqVector(docNum, "contents");
        if (tfv == null) {
        // ignore empty fields
        continue;
        }
        String terms[] = tfv.getTerms();
        int termCount = terms.length;
        int freqs[] = tfv.getTermFrequencies();

        for (int t=0; t < termCount; t++) {
        System.out.println(terms[t] + " " + freqs[t]);
        }
    }

lucene 中是否有任何内置函数可以让我这样做?

Is there any buit-in function in lucene for me to do that?

没有人帮忙，我自己做了:

Nobody helped, and I did it by myself:

    Directory dir = FSDirectory.open(new File(indexDir));
    IndexReader ir = IndexReader.open(dir);

    int docNum;
    for (docNum = 0; docNum<ir.numDocs(); docNum++) {
        TermFreqVector tfv = ir.getTermFreqVector(docNum, "title");
        if (tfv == null) {
                // ignore empty fields
                continue;
        }
        String tterms[] = tfv.getTerms();
        int termCount = tterms.length;
        int freqs[] = tfv.getTermFrequencies();

        for (int t=0; t < termCount; t++) {
            double idf = ir.numDocs()/ir.docFreq(new Term("title", tterms[t]));
            System.out.println(tterms[t] + " " + freqs[t]*Math.log(idf));
        }
    }

有什么方法可以找到每个词条的ID号吗?

is there any way to find the ID number of each term?

没有人帮忙，我自己又做了一次:

Nobody helped, and I did it by myself again:

    List list = new LinkedList();
    terms = null;
    try
    {
        terms = ir.terms(new Term("title", ""));
        while ("title".equals(terms.term().field()))
        {
        list.add(terms.term().text());
        if (!terms.next())
            break;
        }
    }
    finally
    {
        terms.close();
    }
    int docNum;
    for (docNum = 0; docNum<ir.numDocs(); docNum++) {
        TermFreqVector tfv = ir.getTermFreqVector(docNum, "title");
        if (tfv == null) {
                // ignore empty fields
                continue;
        }
        String tterms[] = tfv.getTerms();
        int termCount = tterms.length;
        int freqs[] = tfv.getTermFrequencies();

        for (int t=0; t < termCount; t++) {
            double idf = ir.numDocs()/ir.docFreq(new Term("title", tterms[t]));
            System.out.println(Collections.binarySearch(list, tterms[t]) + " " + tterms[t] + " " + freqs[t]*Math.log(idf));
        }
    }

推荐答案

你可能找不到 tf-idf 向量.但正如您已经完成的那样，您可以手动计算 IDF.最好使用 DefaultSimilarity(或您使用的任何相似性实现)为您计算它.

You'll probably not found a tf-idf vector. But as you've already done, you can calculate IDF by hand. It is probably better to use the DefaultSimilarity (or whatever Similarity implementation you are using) to calculate it for you.

关于 Term ID，我认为目前你不能.至少直到 Lucene 4.0，见 这个.

Regarding Term ID, I think currently you can't. At least not until Lucene 4.0, see this.

这篇关于用 lucene 提取 tf-idf 向量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！