将NLTK停用词与scikit-learn的TfidfVectorizer一起使用时的Unicode警告 [英] Unicode Warning when using NLTK stopwords with TfidfVectorizer of scikit-learn
本文介绍了将NLTK停用词与scikit-learn的TfidfVectorizer一起使用时的Unicode警告的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我正在尝试使用scikit-learn中的Tf-idf Vectorizer,并使用NLTK中的西班牙语停用词:
I am trying to use the Tf-idf Vectorizer from scikit-learn, using the spanish stopwords from NLTK:
from nltk.corpus import stopwords
vectorizer = TfidfVectorizer(stop_words=stopwords.words("spanish"))
问题是我收到以下警告:
The problem is that I get the following warning:
/home/---/.virtualenvs/thesis/local/lib/python2.7/site-packages/sklearn/feature_extraction/text.py:122: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
tokens = [w for w in tokens if w not in stop_words]
有解决此问题的简便方法吗?
Is there an easy way to solve this issue?
推荐答案
实际上,这个问题比我想象的要容易解决.这里的问题是NLTK不返回unicode对象,而是str对象.因此,在使用它们之前,我需要从utf-8对其进行解码:
Actually the problem was more easy to solve than I thought. The issue here is that NLTK does not return unicode object, but str objects. So I needed to decode them from utf-8 before using them:
stopwords = [word.decode('utf-8') for word in stopwords.words('spanish')]
这篇关于将NLTK停用词与scikit-learn的TfidfVectorizer一起使用时的Unicode警告的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文