如何使TF-IDF矩阵密集? [英] How to make TF-IDF matrix dense?

问题描述

我正在使用 TfidfVectorizer 将原始文档的集合转换为TF-IDF功能矩阵，然后我计划将其输入到k-means算法(我将实现)中.在该算法中，我将必须计算质心(商品类别)和数据点(商品)之间的距离.我将使用欧几里得距离，因此在我的情况下，我需要这两个实体具有相同的维数max_features.这是我所拥有的:

I am using TfidfVectorizer to convert a collection of raw documents to a matrix of TF-IDF features, which I then plan to input into a k-means algorithm (which I will implement). In that algorithm I will have to compute distances between centroids (categories of articles) and data points (articles). I am going to use Euclidean distance, so I need these two entities to be of same dimension, in my case max_features. Here is what I have:

tfidf = TfidfVectorizer(max_features=10, strip_accents='unicode', analyzer='word', stop_words=stop_words.extra_stopwords, lowercase=True, use_idf=True)
X = tfidf.fit_transform(data['Content']) # the matrix articles x max_features(=words)
for i, row in enumerate(X):
    print X[i]

但是X似乎是一个稀疏(?)矩阵，因为输出为:

However X seems to be a sparse(?) matrix, since the output is:

  (0, 9)    0.723131915847
  (0, 8)    0.090245047798
  (0, 6)    0.117465276892
  (0, 4)    0.379981697363
  (0, 3)    0.235921470645
  (0, 2)    0.0968780456528
  (0, 1)    0.495689001273

  (0, 9)    0.624910843051
  (0, 8)    0.545911131362
  (0, 7)    0.160545991411
  (0, 5)    0.49900042174
  (0, 4)    0.191549050212

  ...

我认为在哪里(0, col)指出矩阵中的列索引，实际上就像一个数组，其中每个单元格都指向一个列表.

Where I think the (0, col) states the column index in the matrix, which actually like an array, where every cell points to a list.

如何将该矩阵转换为密集矩阵(以便每一行具有相同的列数)?

>print type(X)
<class 'scipy.sparse.csr.csr_matrix'>

推荐答案

这应该很简单:

dense = X.toarray()

TfIdfVectorizer.fit_transform()返回一个SciPy csr_matrix() (压缩稀疏行矩阵)，它具有toarray()方法.在SciPy中，稀疏矩阵有几种格式，但是它们都有一个 .toarray() 方法.

TfIdfVectorizer.fit_transform() is returning a SciPy csr_matrix() (Compressed Sparse Row Matrix), which has a toarray() method just for this purpose. There are several formats of sparse matrices in SciPy, but they all have a .toarray() method.

请注意，对于大型矩阵，相比于稀疏矩阵，这将占用大量内存，因此通常，这是使它尽可能长时间保持稀疏的好方法.

Note that for a large matrix, this will use a tremendous amount of memory compared to a sparse matrix, so generally it's a good approach to leave it sparse for as long as possible.

这篇关于如何使TF-IDF矩阵密集?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！