scipy.sparse.linalg.sp为Linux系统上的大型稀疏矩阵解决了令人惊讶的行为 [英] scipy.sparse.linalg.spsolve surprising behaviour for large sparse matrices on Linux systems
问题描述
我正在计算线性系统Ax = b的解，其中A具有一个大的稀疏矩阵(对于相关的密集矩阵，通常为200,000行和几列)，而b是大约100列的稀疏矩阵.
I am computing the solution of a linear system Ax=b with A a large (typically 200,000 lines and columns for the associated dense matrix) sparse matrix and b a sparse matrix of about 100 columns.
当我在Windows系统(Python 2.7 ，scipy 0.14.0 )上运行代码时，以下命令
When I run my code on Windows systems (Python 2.7, scipy 0.14.0), the following command
from scipy.sparse.linalg import spsolve
...
Temp = spsolve(A.tocsc(),b.tocsc())
运行平稳，需要大约7 GB的ram内存.
runs smoothly and requires about 7 GB of ram memory.
在Linux系统(相同的CPU，相同的RAM内存量:64 GB，Linux Mint 17.3 <)上，使用完全相同的矩阵和完全相同的矩阵运行/strong>，python 2.7 ，scipy 0.13.3 )需要超过20 GB的内存，并且崩溃并显示以下错误消息:
Running the exact same code with the exact same matrices on Linux systems (same CPU, same amount of RAM memory: 64 GB, Linux Mint 17.3, python 2.7, scipy 0.13.3) requires more than 20 GB of ram memory and it crashes with the following error message:
<function umfpack_di_numeric at ...> failed with UMFPACK_ERROR_out_of_memory(请参阅 1 )
由于此错误取决于操作系统，因此我排除了有关矩阵 A 和 b 的任何问题(与某些提到的解决方案
Because this error is os dependent, I ruled out any issue regarding the matrices A and b (contrary to some of the mentioned solutions in this post), and I am trying to find a fix specific to Linux... But I do not know where to start... Does anyone would have any idea of what is going on ? And why would such a problem be specific to Linux systems ?
请在下面找到完整的错误消息:
Please find below the full error message:
Exception in Tkinter callback
Traceback (most recent call last):
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1489, in __call__
return self.func(*args)
File "...", line 1533, in mmvConstruction
...
File "...", line 1555, in modes_cb
Temp = spsolve(k[inter][:,inter].tocsc(),k[inter][:,exter].tocsc())
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/dsolve/linsolve.py", line 151, in spsolve
Afactsolve = factorized(A)
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/dsolve/linsolve.py", line 352, in factorized
umf.numeric(A)
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/dsolve/umfpack/umfpack.py", line 450, in numeric
umfStatus[status]))
RuntimeError:<function umfpack_di_numeric at ...> failed with UMFPACK_ERROR_out_of_memory
Exception in Tkinter callback
Traceback (most recent call last):
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1489, in __call__
return self.func(*args)
File "...", line 1533, in mmvConstruction
...
File "...", line 1555, in modes_cb
Temp = spsolve(k[inter][:,inter].tocsc(),k[inter][:,exter].tocsc())
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/dsolve/linsolve.py", line 151, in spsolve
Afactsolve = factorized(A)
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/dsolve/linsolve.py", line 352, in factorized
umf.numeric(A)
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/dsolve/umfpack/umfpack.py", line 450, in numeric
umfStatus[status]))
RuntimeError:<function umfpack_di_numeric at ...> failed with UMFPACK_ERROR_out_of_memory
更新:仍在尝试寻找解决方案... Linux Mint上BLAS的最新版本似乎已经很旧了:1.8.2.在Windows上，我使用BLAS 1.9.1.使用test_numpy.py文件时，可在此处找到: https://gist.github. com/osdf/3842524#file-test_numpy-py 我注意到Linux和Windows之间存在非常明显的区别: Linux :版本1.8.2，maxint 922337203685477575807，点号:0.76 s- Windows :版本1.9.1，maxint 2147483647，点:0.037 s.我正在研究Linux上的OPENBLAS是否可以解决此问题...
Update: still trying to find out a solution... it seems that the latest version of BLAS on Linux Mint is quite old: 1.8.2. On Windows, I use BLAS 1.9.1. When using the test_numpy.py file available here: https://gist.github.com/osdf/3842524#file-test_numpy-py I notice very significant differences between Linux and Windows: Linux : version 1.8.2, maxint 9223372036854775807, dot: 0.76 s - Windows : version 1.9.1, maxint 2147483647, dot: 0,037 s. I am investigating if OPENBLAS on Linux could be a solution to this problem...
更新2:我意识到问题可能与硬件有关.确实，一台旧的PC在相同的Linux Mint发行版(Rosa 17.3)上具有完全相同的库，可以提供令人满意的结果.第一次更新中提到的基准在该旧PC上提供了 Linux :版本1.8.2，maxint 9223372036854775807，点:0,054 s.
Update 2: I realized that the problem may be hardware related. Indeed, an older PC, with the exact same libraries on the same Linux Mint distribution (Rosa 17.3) provides much more satisfying results. The benchmark mentioned in the first update gives on this old PC: Linux : version 1.8.2, maxint 9223372036854775807, dot: 0,054 s.
推荐答案
好吧，经过深入的研究，我现在确信我遇到的问题与 Linux Mint(Rosa 17.3)有关>可能无法针对最新的处理器进行优化.
Well, after thorough investigations, I am now convinced that the issue I had is related to the fact that Linux Mint (Rosa 17.3) may not be optimized for the most recent processors.
我在帖子更新中提到的比较结果表明该软件安装正确.然后，我在PC上安装了 Fedora 23 ，并按顺序安装:
The comparisons I mentioned in the updates of my post underlined that the software installation was correct. I then installed Fedora 23 on my PC, installing sequentially:
- libblas
- python
- python-scipy
然后，我使用完全相同的矩阵运行了完全相同的代码，并且没有任何问题:RAM消耗限制为大约7 GB，与Windows系统上观察到的情况类似.
Then I ran the exact same code with the exact same matrices and there was no issue whatsoever: RAM consumption was limited to about 7 GB, similarly to what was observed on Windows systems.
这篇关于scipy.sparse.linalg.sp为Linux系统上的大型稀疏矩阵解决了令人惊讶的行为的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
