Python多处理池挂在连接处? [英] Python multiprocessing pool hangs at join?

问题描述

我正在尝试在多个文件上并行运行一些python代码.构造基本上是:

I'm trying to run some python code on several files in parallel. The construct is basically:

def process_file(filename, foo, bar, baz=biz):
    # do stuff that may fail and cause exception

if __name__ == '__main__':
    # setup code setting parameters foo, bar, and biz

    psize = multiprocessing.cpu_count()*2
    pool = multiprocessing.Pool(processes=psize)

    map(lambda x: pool.apply_async(process_file, (x, foo, bar), dict(baz=biz)), sys.argv[1:])
    pool.close()
    pool.join()

我以前使用过pool.map做类似的事情，并且效果很好，但是我似乎不能在这里使用它，因为pool.map不允许(似乎)允许我传递额外的参数(并且使用lambda进行操作将无法正常工作，因为无法将lambda编组).

I've previously used pool.map to do something similar and it worked great, but I can't seem to use that here because pool.map doesn't (appear to) allow me to pass in extra arguments (and using lambda to do it won't work because lambda can't be marshalled).

因此，现在我正尝试直接使用apply_async()使事情正常进行.我的问题是代码似乎挂起并且永远不会退出.一些文件例外地失败了，但是我不明白为什么会导致连接失败/挂起?有趣的是，如果所有文件均未因异常而失败，则它会干净地退出.

So now I'm trying to get things to work using apply_async() directly. My issue is that the code seems to hang and never exit. A few of the files fail with an exception, but i don't see why what would cause join to fail/hang? Interestingly if none of the files fail with an exception, it does exit cleanly.

我想念什么?

当函数(从而导致一个工作程序)失败时，我看到此异常:

When the function (and thus a worker) fails, I see this exception:

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 376, in _handle_results
    task = get()
TypeError: ('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())

如果我什至看到其中之一，则进程父进程将永远挂起，永远不会收获子进程并退出.

If i see even one of these, the process parent process hangs forever, never reaping the children and exiting.

推荐答案

很抱歉回答我自己的问题，但是我至少找到了一种解决方法，以防万一其他人遇到类似的问题，我想在此发布.我会接受任何更好的答案.

Sorry to answer my own question, but I've found at least a workaround so in case anyone else has a similar issue I want to post it here. I'll accept any better answers out there.

我认为问题的根源是 http://bugs.python.org/issue9400 .这告诉我两件事:

I believe the root of the issue is http://bugs.python.org/issue9400 . This tells me two things:

• 我不疯，我真正想做的事应该可以奏效
• 至少在python2中，很难(即使不是不可能)将异常"腌制回父进程.简单的方法可以工作，而其他许多方法则不能.

就我而言，我的工作人员功能正在启动一个存在段错误的子流程.返回的CalledProcessError异常是不可腌制的.由于某种原因，这会使父对象中的pool对象外出吃午饭，而不是从对join()的调用中返回.

In my case, my worker function was launching a subprocess that was segfaulting. This returned CalledProcessError exception, which is not pickleable. For some reason, this makes the pool object in the parent go out to lunch and not return from the call to join().

在我的特定情况下，我不在乎异常是什么.最多我想记录下来并继续前进.为此，我只需将我的顶级worker函数包装在try/except子句中.如果工作程序引发任何异常，则会在尝试返回到父进程之前对其进行捕获，并进行记录，然后由于该工作进程不再尝试通过其发送异常，因此正常退出.见下文:

In my particular case, I don't care what the exception was. At most I want to log it and keep going. To do this, I simply wrap my top worker function in a try/except clause. If the worker throws any exception, it is caught before trying to return to the parent process, logged, and then the worker process exits normally since it's no longer trying to send the exception through. See below:

def process_file_wrapped(filenamen, foo, bar, baz=biz):
    try:
        process_file(filename, foo, bar, baz=biz)
    except:
        print('%s: %s' % (filename, traceback.format_exc()))

然后，我有一个初始的地图函数，叫做process_file_wrapped()，而不是原来的那个.现在，我的代码可以按预期工作.

Then, I have my initial map function call process_file_wrapped() instead of the original one. Now my code works as intended.

这篇关于Python多处理池挂在连接处?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！