多重处理:NULL结果，PyObject_Call中没有错误 [英] Multiprocessing : NULL result without error in PyObject_Call

问题描述

这是我使用多重处理的示例程序.使用multiprocessing.Process完成计算，并使用multiprocessing.Queue收集结果.

Here is a sample program where I use multiprocessing. The calculations are done with multiprocessing.Process and the results are collected using multiprocessing.Queue.

#THIS PROGRAM RUNS WITH ~40Gb RAM. (you can reduce a,b,c for less RAM 
#but then it works for smaller values)
#PROBLEM OCCURS ONLY FOR HUGE DATA.   
from numpy import *
import multiprocessing as mp

a = arange(0, 3500, 5)
b = arange(0, 3500, 5)
c = arange(0, 3500, 5)  
a0 = 540. #random values
b0 = 26.
c0 = 826.
def rand_function(a, b, c, a0, b0, c0):
    Nloop = 100.
    def loop(Nloop, out):
        res_total = zeros((700, 700, 700), dtype = 'float') 
        n = 1
        while n <= Nloop:
            rad = sqrt((a-a0)**2 + (b-b0)**2 + (c-c0)**2)
            res_total += rad
            n +=1 
        out.put(res_total)
    out = mp.Queue() 
    jobs = []
    Nprocs = mp.cpu_count()
    print "No. of processors : ", Nprocs
    for i in range(Nprocs):
        p = mp.Process(target = loop, args=(Nloop/Nprocs, out)) 
        jobs.append(p)
        p.start()

    final_result = zeros((700, 700, 700), dtype = 'float')

    for i in range(Nprocs):
        final_result = final_result + out.get()

    p.join()
test = rand_function(a,b,c,a0, b0, c0)

这是错误消息:

Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 266, in _feed
    send(obj)
SystemError: NULL result without error in PyObject_Call

我在此处读到这是一个错误.但是我听不懂. 谁能告诉我使用多处理计算海量数据的任何出路吗?

I read here that it is a bug. But I am unable to understand. Can anyone please tell me any way out to calculate huge data using multiprocessing?

非常感谢

推荐答案

该错误报告引用了您的状态，指出多处理模块无法将大量参数推入子进程.

The bug report your reference states that multiprocessing module is unable to push huge arguments to subprocess.

原因是它需要腌制这些参数并将腌制的blob存储在内存中的某个位置.

The reason is that it needs to pickle these arguments and store the pickled blob somewhere in memory.

但是，您不需要将数组作为参数传递.

You, however, don't need to pass arrays as arguments.

可能的原因:

• 将闭包loop作为目标传递
• 将mp.Queue()作为参数传递

• passing a closure loop as a target
• passing mp.Queue() as argument

请参阅 http://stevenengelhardt.com/2013 /01/16/python-multiprocessing-module-and-closures/关于将闭包转换为类.

Please see http://stevenengelhardt.com/2013/01/16/python-multiprocessing-module-and-closures/ about converting your closure to a class.

设置完全状态，然后再控制多处理.

Set up full state before you give control to multiprocessing.

这篇关于多重处理:NULL结果，PyObject_Call中没有错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！