ValueError在SciPy中采用两个稀疏矩阵的点积 [英] ValueError taking dot product of two sparse matrices in SciPy

问题描述

我正在尝试取两个近似为lil_matrix稀疏矩阵的点积. 100,000 x 50,000和50,000 x 100,000.

I'm trying to take the dot product of two lil_matrix sparse matrices that are approx. 100,000 x 50,000 and 50,000 x 100,000 respectively.

from scipy import sparse
a = sparse.lil_matrix((100000, 50000))
b = sparse.lil_matrix((50000, 100000))

c = a.dot(b)

并收到此错误:

 File "/usr/lib64/python2.6/site-packages/scipy/sparse/base.py", line 211, in dot
 return self * other
 File "/usr/lib64/python2.6/site-packages/scipy/sparse/base.py", line 247, in __mul__
 return self._mul_sparse_matrix(other)
 File "/usr/lib64/python2.6/site-packages/scipy/sparse/base.py", line 300, in      _mul_sparse_matrix
 return self.tocsr()._mul_sparse_matrix(other)
 File "/usr/lib64/python2.6/site-packages/scipy/sparse/compressed.py", line 290, in _mul_sparse_matrix
 indices = np.empty(nnz, dtype=np.intc)
 ValueError: negative dimensions are not allowed

关于可能发生的情况的任何想法-在具有约64GB RAM的计算机上运行该代码，并在执行点时使用约13GB内存.

Any ideas on what might be happening - running this on a machine with about 64GB of ram, and using about 13GB when executing the dot.

推荐答案

这是错误的错误消息，但问题"很简单，就是您得到的矩阵太大(具有太多非零元素，而不是维数) ).

This is a bad error message, but the "problem" quite simply is that your resulting matrix would be too big (has too many nonzero elements, not its dimension).

Scipy使用int32来存储稀疏格式的indptr和indices.这意味着您的稀疏矩阵不能(大约)具有2 ^ 31个非零元素.如果这不只是玩具问题，也许您可​​以将scipy中的代码更改为使用int64或uint32.但是也许使用稀疏矩阵并不是解决这个问题的最佳解决方案吗?

Scipy uses int32 to store indptr and indices for the sparse formats. This means that your sparsematrix cannot have more then (approximatly) 2^31 nonzero elements. Maybe you could change the code in scipy to use int64 or uint32, if this is not just a toy problem anyways. But maybe the use of sparse matrixes is not the best solution for solving this anyways?

编辑:此问题已在新的scipy版本AFIAK中解决.

This is solved in the new scipy versions AFIAK.

这篇关于ValueError在SciPy中采用两个稀疏矩阵的点积的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！