Python多处理脚本似乎无错误地冻结 [英] Python Multiprocessing Script Freezes Seemingly Without Error

问题描述

我正在尝试使用multiprocessing包并行调用一个函数(我们称其为myfunc)，特别是使用pool.map即pool.map(myfunc, myarglist).当我简单地循环使用myarglist而不使用multiprocessing时，没有错误，应该是这种情况，因为myfunc中的所有操作都在try块内调用.但是，当我使用pool.map调用该函数时，脚本总是停止运行，即，它停止打印"myfunc done！".我的函数中的语句，并且进程停止使用CPU，但是它从不返回resultlist.我正在Ubuntu 12.04的终端上运行python 2.7.可能导致这种情况发生的原因以及我该如何解决/解决问题?

cpu_count = int(multiprocessing.cpu_count())
pool = Pool(processes = cpu_count)
resultlist = pool.map(myfunc, myarglist)
pool.close()

更新 使用多重处理时，一个问题可能是对象的大小，如果您认为可能是个问题，请参阅此答案.正如答案所指出的那样:如果此[解决方案]不起作用，则您从函数中返回的内容可能不会被腌制，因此无法正确地通过队列."多重处理通过对对象进行腌制来在它们之间传递对象.事实证明，我的一两个对象从BeautifulSoup 解决方案

检查是否已启动所有进程.这将有助于您对其进行调试.还应在代码末尾添加Pool.join(). /p>

这是示例代码

def start_process():
    print 'Starting', multiprocessing.current_process().name

if __name__ == '__main__':

    pool_size =2
    pool = multiprocessing.Pool(processes=pool_size,
                                initializer=start_process,
                                )

    pool_outputs = pool.map(function_name,argument_list)
    pool.close() # no more tasks
    pool.join()  # wrap up current tasks

I am trying to use the multiprocessing package to call a function (let's call it myfunc) in parallel, specifically using pool.map i.e. pool.map(myfunc, myarglist). When I simply loop over myarglist without using multiprocessing there are no errors, which should be the case because all operations in myfunc are called within a try block. However, when I call the function using pool.map the script invariably stops running, i.e. it stop printing a "myfunc done!" statement within my function and the processes stop using the CPUs, but it never returns resultlist. I am running python 2.7 from the terminal in ubuntu 12.04. What could cause this to occur and how should I fix/troubleshoot the problem?

cpu_count = int(multiprocessing.cpu_count())
pool = Pool(processes = cpu_count)
resultlist = pool.map(myfunc, myarglist)
pool.close()

Update One issue when using multiprocessing can be the size of the object, if you think that may be a problem see this answer. As the answer notes "If this [solution] doesn't work, maybe the stuff you're returning from your functions is not pickleable, and therefore unable to make it through the Queues properly." Multiprocessing passes objects between processes by pickling them. It turns out that one or two of my objects had soup from BeautifulSoup that would not pickle.

解决方案

Check whether all the processes are started or not.This will help you to debug it.Also add Pool.join() at the end of your code.

This is a sample code

def start_process():
    print 'Starting', multiprocessing.current_process().name

if __name__ == '__main__':

    pool_size =2
    pool = multiprocessing.Pool(processes=pool_size,
                                initializer=start_process,
                                )

    pool_outputs = pool.map(function_name,argument_list)
    pool.close() # no more tasks
    pool.join()  # wrap up current tasks

这篇关于Python多处理脚本似乎无错误地冻结的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！