使用numpy进行多处理会使Python在OSX上意外退出 [英] Multiprocessing with numpy makes Python quit unexpectedly on OSX

问题描述

我遇到了一个问题，当使用numpy运行多处理时，Python意外退出.我已经隔离了这个问题，因此我现在可以确认在运行以下代码时，多处理可以完美地工作:

I've run into a problem, where Python quits unexpectedly, when running multiprocessing with numpy. I've isolated the problem, so that I can now confirm that the multiprocessing works perfect when running the code stated below:

import numpy as np
from multiprocessing import Pool, Process
import time
import cPickle as p

def test(args):
    x,i = args
    if i == 2:
        time.sleep(4)
    arr = np.dot(x.T,x)
    print i

if __name__ == '__main__':
    x = np.random.random(size=((2000,500)))
    evaluations = [(x,i) for i in range(5)]
    p = Pool()
    p.map_async(test,evaluations)
    p.close()
    p.join()

当我尝试评估下面的代码时，会出现问题.这使Python意外退出:

The problem occurs when I try to evaluate the code below. This makes Python quit unexpectedly:

import numpy as np
from multiprocessing import Pool, Process
import time
import cPickle as p

def test(args):
    x,i = args
    if i == 2:
        time.sleep(4)
    arr = np.dot(x.T,x)
    print i

if __name__ == '__main__':
    x = np.random.random(size=((2000,500)))
    test((x,4)) # Added code
    evaluations = [(x,i) for i in range(5)]
    p = Pool()
    p.map_async(test,evaluations)
    p.close()
    p.join()

请帮助某人.我愿意接受所有建议.谢谢.注意:我尝试了两台不同的机器，并且会出现相同的问题.

Please help someone. I'm open to all suggestions. Thanks. Note: I have tried two different machines and the same problem occurs.

推荐答案

我想出了解决此问题的方法.在初始化多处理实例之前，将Numpy与BLAS一起使用时会发生此问题.我的解决方法只是将Numpy代码(运行BLAS)放入单个进程中，然后再运行多处理实例.这不是一种很好的编码样式，但是它可以工作.请参见下面的示例:

I figured out a workaround to the problem. The problem occurs when Numpy is used together with BLAS before initializing a multiprocessing instance. My workaround is simply to put the Numpy code (running BLAS) into a single process and then running the multiprocessing instances afterwards. This is not a good coding style, but it works. See example below:

以下操作将失败-Python将退出:

Following will fail - Python will quit:

import numpy as np
from multiprocessing import Pool, Process

def test(x):
    arr = np.dot(x.T,x) # On large matrices, this calc will use BLAS.

if __name__ == '__main__':
    x = np.random.random(size=((2000,500))) # Random matrix
    test(x)
    evaluations = [x for _ in range(5)]
    p = Pool()
    p.map_async(test,evaluations) # This is where Python will quit, because of the prior use of BLAS.
    p.close()
    p.join()

以下将会成功:

import numpy as np
from multiprocessing import Pool, Process

def test(x):
    arr = np.dot(x.T,x) # On large matrices, this calc will use BLAS.

if __name__ == '__main__':
    x = np.random.random(size=((2000,500))) # Random matrix
    p = Process(target = test,args = (x,))
    p.start()
    p.join()
    evaluations = [x for _ in range(5)]
    p = Pool()
    p.map_async(test,evaluations)
    p.close()
    p.join()

这篇关于使用numpy进行多处理会使Python在OSX上意外退出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！