在python中计算没有停用词的tfidf矩阵 [英] calculate tfidf matrix without stop words in python
本文介绍了在python中计算没有停用词的tfidf矩阵的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我正在尝试计算一个没有停用词的 tfidf
矩阵.这是我的代码:
I'm trying to calculate a tfidf
matrix without the stopwords. This is my code:
def removeStopWords(documents):
stop_words = set(stopwords.words('italian'))
english_stop_words = set(stopwords.words('english'))
stop_words.update(list(set(english_stop_words)))
for d in documents:
document = d['document']
word_tokens = word_tokenize(document)
filtered_sentence = ''
for w in word_tokens:
if not inStopwords(w, stop_words):
filtered_sentence = w + ' ' + filtered_sentence
d['document'] = filtered_sentence[:-1]
return calculateTFIDF(documents)
def calculateTFIDF(corpus):
tfidf = TfidfVectorizer()
x = tfidf.fit_transform(corpus)
df_tfidf = pd.DataFrame(x.toarray(), columns=tfidf.get_feature_names())
return {c: s[s > 0] for c, s in zip(df_tfidf, df_tfidf.T.values)}
但是当我返回矩阵(形式为 {word:value}
)时,它还包含一些停用词,如 when
或 il
.我该如何解决?谢谢
But when I return the matrix ( with the form {word:value}
), it contains also some stopwords like when
or il
. How can I resolve ? Thanks
推荐答案
有更好的方法可以去除 TF-IDF 计算的停用词.TfidfVectorizer
有一个参数 stop_words
,您可以在其中传递要排除的单词集合.
There are better methods to remove stopwords for TF-IDF calculation. The TfidfVectorizer
has a parameter stop_words
where you can pass a collection of words to exclude.
from nltk.corpus import stopwords
from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
documents = ['I went to the barbershop when my hair was long.', 'The barbershop was closed.']
# create set of stopwords to remove
stop_words = set(stopwords.words('italian'))
english_stop_words = set(stopwords.words('english'))
stop_words.update(english_stop_words)
# check if word in stop words
print('when' in stop_words) # True
print('il' in stop_words) # True
# else add word to the set
print('went' in stop_words) # False
stop_words.add('went')
# create tf-idf from original documents
tfidf = TfidfVectorizer(stop_words=stop_words)
x = tfidf.fit_transform(documents)
df_tfidf = pd.DataFrame(x.toarray(), columns=tfidf.get_feature_names())
print({c: s[s > 0] for c, s in zip(df_tfidf, df_tfidf.T.values)})
# {'barbershop': array([0.44943642, 0.57973867]), 'closed': array([0.81480247]), 'hair': array([0.6316672]), 'long': array([0.6316672])}
这篇关于在python中计算没有停用词的tfidf矩阵的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文