与NumPy不兼容的多处理 [英] Multiprocessing incompatible with NumPy

问题描述

我正在尝试使用多处理程序运行一个简单的测试.在导入numpy之前，该测试效果良好(即使程序中未使用它).这是代码:

I am trying to run a simple test using multiprocessing. The test works well until I import numpy (even though it is not used in the program). Here is the code:

from multiprocessing import Pool
import time
import numpy as np #this is the problematic line


def CostlyFunc(N):
    """"""
    tstart = time.time()
    x = 0
    for i in xrange(N):
        for j in xrange(N):
            if i % 2: x += 2
            else: x -= 2       
    print "CostlyFunc : elapsed time %f s" % (time.time() - tstart)
    return x

#serial application
ResultList0 = []
StartTime = time.time()
for i in xrange(3):
    ResultList0.append(CostlyFunc(5000))
print "Elapsed time (serial) : ", time.time() - StartTime


#multiprocessing application
StartTime = time.time()
pool = Pool()
asyncResult = pool.map_async(CostlyFunc, [5000, 5000, 5000])
ResultList1 = asyncResult.get()
print "Elapsed time (multiporcessing) : ", time.time() - StartTime

如果我不导入numpy，则结果为:

If I don't import numpy the result is:

CostlyFunc : elapsed time 2.866265 s
CostlyFunc : elapsed time 2.793213 s
CostlyFunc : elapsed time 2.794936 s
Elapsed time (serial) :  8.45455098152
CostlyFunc : elapsed time 2.889815 s
CostlyFunc : elapsed time 2.891556 s
CostlyFunc : elapsed time 2.898898 s
Elapsed time (multiporcessing) :  2.91595196724

总的经过时间类似于1个过程所需的时间，这意味着计算已并行化.如果我导入numpy，结果将变为:

The total elapsed time is similar to the time required for 1 process, meaning that the computation has been parallelized. If I do import numpy the result becomes :

CostlyFunc : elapsed time 2.877116 s
CostlyFunc : elapsed time 2.866778 s
CostlyFunc : elapsed time 2.860894 s
Elapsed time (serial) :  8.60492110252
CostlyFunc : elapsed time 8.450145 s
CostlyFunc : elapsed time 8.473006 s
CostlyFunc : elapsed time 8.506402 s
Elapsed time (multiporcessing) :  8.55398178101

串行和多处理方法所用的总时间相同，因为仅使用一个内核.显然问题出在numpy.我的多处理版本与NumPy之间是否可能不兼容?

The total time elapsed is the same for both serial and multiprocessing methods because only one core is used. It is clear that the problem comes from numpy. Is it possible that I have an incompatibility between my versions of multiprocessing and NumPy?

我目前在Linux上使用Python2.7，NumPy 1.6.2和多处理0.70a1

I am currently using Python2.7, NumPy 1.6.2 and multiprocessing 0.70a1 on linux

推荐答案

(如果措辞不佳或未正确引用，请先发表帖子)

(First Post sorry if it is not well formulated or alligned)

通过将MKL_NUM_THREADS设置为1，可以停止Numpy使用多线程.

You can stop Numpy to use multithreading by seting the MKL_NUM_THREADS to 1

在我使用过的debian下:

Under debian I used:

export MKL_NUM_THREADS=1

相关stackoverflow帖子的来源: Python:如何停止numpy来自多线程?

Source from related stackoverflow post: Python: How do you stop numpy from multithreading?

结果:

user@pc:~/tmp$ python multi.py
CostlyFunc : elapsed time 3.847009 s
CostlyFunc : elapsed time 3.253226 s
CostlyFunc : elapsed time 3.415734 s
Elapsed time (serial) :  10.5163660049
CostlyFunc : elapsed time 4.218424 s
CostlyFunc : elapsed time 5.252429 s
CostlyFunc : elapsed time 4.862513 s
Elapsed time (multiporcessing) :  9.11713695526

user@pc:~/tmp$ export MKL_NUM_THREADS=1

user@pc:~/tmp$ python multi.py
CostlyFunc : elapsed time 3.014677 s
CostlyFunc : elapsed time 3.102548 s
CostlyFunc : elapsed time 3.060915 s
Elapsed time (serial) :  9.17840886116
CostlyFunc : elapsed time 3.720322 s
CostlyFunc : elapsed time 3.950583 s
CostlyFunc : elapsed time 3.656165 s
Elapsed time (multiporcessing) :  7.399310112

我不确定是否有帮助，因为我猜您最终希望numpy并行运行，也许会尝试调整numpy到您计算机的线程数.

I am not sure if that helps because I guess eventually you want numpy to run in parallel maybe try to adjust the number of threads for numpy to your machine.

这篇关于与NumPy不兼容的多处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！