如何从文档术语矩阵中提取词频? [英] How to extract word frequency from document-term matrix?
问题描述
我正在使用Python进行LDA分析.我用下面的代码创建了一个文档术语矩阵
I am doing LDA analysis with Python. And I used the following code to create a document-term matrix
corpus = [dictionary.doc2bow(text) for text in texts].
有没有简单的方法可以计算整个语料库中单词的出现频率.由于我确实有一个字典,它是一个术语ID列表,所以我认为我可以将频率词与术语ID匹配.
Is there any easy ways to count the word frequency over the whole corpus. Since I do have the dictionary which is a term-id list, I think I can match the word frequency with term-id.
推荐答案
您可以使用nltk
来计算字符串texts
You can use nltk
in order to count word frequency in string texts
from nltk import FreqDist
import nltk
texts = 'hi there hello there'
words = nltk.tokenize.word_tokenize(texts)
fdist = FreqDist(words)
fdist
将为您提供给定字符串texts
的单词频率.
fdist
will give you word frequency of given string texts
.
但是,您有一个文本列表.一种计算频率的方法是使用scikit-learn
中的CountVectorizer
作为字符串列表.
However, you have a list of text. One way to count frequency is to use CountVectorizer
from scikit-learn
for list of strings.
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
texts = ['hi there', 'hello there', 'hello here you are']
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)
freq = np.ravel(X.sum(axis=0)) # sum each columns to get total counts for each word
此freq
将对应于字典vectorizer.vocabulary_
import operator
# get vocabulary keys, sorted by value
vocab = [v[0] for v in sorted(vectorizer.vocabulary_.items(), key=operator.itemgetter(1))]
fdist = dict(zip(vocab, freq)) # return same format as nltk
这篇关于如何从文档术语矩阵中提取词频?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!