Scikit-Learn的管道:通过了稀疏矩阵，但是需要密集数据 [英] Scikit-Learn's Pipeline: A sparse matrix was passed, but dense data is required

问题描述

我发现很难理解如何修复我创建的管道(阅读:很大程度上是从教程中粘贴的).是python 3.4.2:

I'm finding it difficult to understand how to fix a Pipeline I created (read: largely pasted from a tutorial). It's python 3.4.2:

df = pd.DataFrame
df = DataFrame.from_records(train)

test = [blah1, blah2, blah3]

pipeline = Pipeline([('vectorizer', CountVectorizer()), ('classifier', RandomForestClassifier())])

pipeline.fit(numpy.asarray(df[0]), numpy.asarray(df[1]))
predicted = pipeline.predict(test)

运行它，我得到:

TypeError: A sparse matrix was passed, but dense data is required. Use X.toarray() to convert to a dense numpy array.

这是pipeline.fit(numpy.asarray(df[0]), numpy.asarray(df[1]))行的.

我已经通过numpy，scipy等尝试了很多解决方案，但是我仍然不知道如何解决它.是的，以前也曾提出过类似的问题，但不是在管道内部. 我必须在哪里申请toarray或todense?

I've experimented a lot with solutions through numpy, scipy, and so forth, but I still don't know how to fix it. And yes, similar questions have come up before, but not inside a pipeline. Where is it that I have to apply toarray or todense?

推荐答案

不幸的是，这两个不兼容. CountVectorizer生成一个稀疏矩阵，RandomForestClassifier需要一个密集矩阵.可以使用X.todense()进行转换.这样做会大大增加您的内存占用.

Unfortunately those two are incompatible. A CountVectorizer produces a sparse matrix and the RandomForestClassifier requires a dense matrix. It is possible to convert using X.todense(). Doing this will substantially increase your memory footprint.

以下是基于 http进行操作的示例代码://zacstewart.com/2014/08/05/pipelines-of-featureunions-of-pipelines.html ，可让您在管道阶段调用.todense().

Below is sample code to do this based on http://zacstewart.com/2014/08/05/pipelines-of-featureunions-of-pipelines.html which allows you to call .todense() in a pipeline stage.

class DenseTransformer(TransformerMixin):

    def fit(self, X, y=None, **fit_params):
        return self

    def transform(self, X, y=None, **fit_params):
        return X.todense()

拥有DenseTransformer后，就可以将其添加为管道步骤.

Once you have your DenseTransformer, you are able to add it as a pipeline step.

pipeline = Pipeline([
     ('vectorizer', CountVectorizer()), 
     ('to_dense', DenseTransformer()), 
     ('classifier', RandomForestClassifier())
])

另一种选择是使用用于稀疏数据的分类器，例如LinearSVC.

Another option would be to use a classifier meant for sparse data like LinearSVC.

from sklearn.svm import LinearSVC
pipeline = Pipeline([('vectorizer', CountVectorizer()), ('classifier', LinearSVC())])

这篇关于Scikit-Learn的管道:通过了稀疏矩阵，但是需要密集数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！