我应该使用tfidf语料库还是仅使用语料库来使用LDA推理文档? [英] should i use tfidf corpus or just corpus to inference documents using LDA?

问题描述

我只是想知道当我们在gensim中使用LDA推断文档时是使用TFIDF语料库还是只是使用语料库

i am just wondering whether its either TFIDF corpus to be used or just corpus to be used when we are inference documents using LDA in gensim

这是一个例子

from gensim import corpora, models
import numpy.random
numpy.random.seed(10)

doc0 = [(0, 1), (1, 1)]
doc1 = [(0,1)] 
doc2 = [(0, 1), (1, 1)]
doc3 = [(0, 3), (1, 1)]

corpus = [doc0,doc1,doc2,doc3]
dictionary = corpora.Dictionary(corpus)

tfidf = models.TfidfModel(corpus)
corpus_tfidf = tfidf[corpus]
corpus_tfidf.save('x.corpus_tfidf')

corpus_tfidf = corpora.MmCorpus.load('x.corpus_tfidf')

lda = models.ldamodel.LdaModel(corpus_tfidf, id2word=dictionary, num_topics=2)

#which one i should use from this   
**corpus_lda = lda[corpus]**          #this one 
**corpus_LDA = lda[corpus_tfidf ]**   #or this one?


corpus_lda.save('x.corpus_lda')

for i,j in enumerate(corpus_lda):
    print j, corpus[i]

推荐答案

根据

According to Gensim's mailing list (last post in particular) the standard procedure would be to use a bag of words corpus. You can use a TF-IDF corpus, but it seems to be unclear what kind of effect this would have.

这篇关于我应该使用tfidf语料库还是仅使用语料库来使用LDA推理文档?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！