乘用Python非常大的二维数组 [英] Multiplying very large 2D-array in Python

问题描述

我在Python繁殖非常大的二维阵列的100倍左右。每个矩阵由 32000x32000 元素。

I have to multiply very large 2D-arrays in Python for around 100 times. Each matrix consists of 32000x32000 elements.

我用 np.dot（X，Y​​），但这需要很长的时间对每个乘法......下面我的code的一个实例

I'm using np.dot(X,Y), but it takes very long time for each multiplication... Below an instance of my code:

import numpy as np

X = None
for i in range(100)
    multiplying = True
    if X == None:
        X = generate_large_2darray()
        multiplying = False
    else:
        Y = generate_large_2darray()

    if multiplying:
        X = np.dot(X, Y)

是否有任何其他方法快得多？

Is there any other method much faster?

更新

下面是显示HTOP界面的屏幕截图。我的python脚本使用只有一个核心。此外，经过3h25m只有4次乘法已经完成。

Here is a screenshot showing the htop interface. My python script is using only one core. Also, after 3h25m only 4 multiplications have been done.

更新2

我试着执行：

import numpy.distutils.system_info as info
info.get_info('atlas')

但我已经收到了：

but I've received:

/home/francescof/.local/lib/python2.7/site-packages/numpy/distutils/system_info.py:564: UserWarning: Specified path /home/apy/atlas/lib is invalid. warnings.warn('Specified path %s is invalid.' % d) {}

所以，我认为它不是配置好。

So, I think it's not well-configured.

反之亦然，关于 BLAS 我刚收到 {} ，没有警告或错误。

Vice versa, regarding blas I just receive {}, with no warnings or errors.

推荐答案

的建议通过的 ali_m 中，使用BLAS库可以加快操作。然而，在我的系统问题是numpy的不好的配置。这里是解决方案：

As suggested by ali_m, the using of a BLAS library can speed up the operations. However, the problem in my system was a bad configuration of numpy. Here is the solution:

1）确保有所需的所有库（你可以使用ATLAS，OpenBLAS等）。我选择ATLAS在我的情况，因为在Ubuntu直接支持。

1) make sure to have all required libraries (you can use ATLAS, OpenBLAS, etc.). I've chosen ATLAS in my case since directly supported in Ubuntu.

命令和apt-get安装libatlas3gf基libatlas基-dev的libatlas-dev的

2）去除任何previous numpy的装置，例如， pypm卸载numpy的（如果你使用的ActivePython）安装了

2) remove any previous numpy installations, e.g., pypm uninstall numpy (if you installed it using ActivePython)

3）使用numpy的PIP重新安装： PIP安装numpy的

3) install again numpy using pip: pip install numpy

4）确保你的Atlas是正确链接：

4) make sure your atlas is correctly linked:

import numpy.distutils.system_info as info
info.get_info('atlas')

ATLAS version 3.8.4 built by buildd on Sat Sep 10 23:12:12 UTC 2011:
   UNAME    : Linux crested 2.6.24-29-server #1 SMP Wed Aug 10 15:58:57 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
   INSTFLG  : -1 0 -a 1
   ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_HAMMER -DATL_CPUMHZ=1993 -DATL_USE64BITS -DATL_GAS_x8664
   F2CDEFS  : -DAdd_ -DF77_INTEGER=int -DStringSunStyle
   CACHEEDGE: 393216
   F77      : gfortran, version GNU Fortran (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1
   F77FLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -Wa,--noexecstack -fPIC -m64
   SMC      : gcc, version gcc (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1
   SMCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -Wa,--noexecstack -fPIC -m64
   SKC      : gcc, version gcc (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1
   SKCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -Wa,--noexecstack -fPIC -m64
{'libraries': ['lapack', 'f77blas', 'cblas', 'atlas'], 'library_dirs': ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base'], 'define_macros': [('ATLAS_INFO', '"\\"3.8.4\\""')], 'language': 'f77', 'include_dirs': ['/usr/include/atlas']}

这篇关于乘用Python非常大的二维数组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！