检索使用multiprocessing.Pool.map启动的进程的退出代码 [英] Retrieve exit code of processes launched with multiprocessing.Pool.map

问题描述

我正在使用python multiprocessing模块并行化一些计算量大的任务. 显而易见的选择是使用Pool worker，然后使用map方法.

I'm using python multiprocessing module to parallelize some computationally heavy tasks. The obvious choice is to use a Pool of workers and then use the map method.

但是，进程可能会失败.例如，它们可能被oom-killer静默杀死.因此，我希望能够检索使用map启动的进程的退出代码.

However, processes can fail. For instance, they may be silently killed for instance by the oom-killer. Therefore I would like to be able to retrieve the exit code of the processes launched with map.

另外，出于记录目的，我希望能够知道为执行Iterable中的每个值而启动的进程的PID.

Additionally, for logging purpose, I would like to be able to know the PID of the process launched to execute each value in the the iterable.

推荐答案

如果您使用的是multiprocessing.Pool.map，则通常对池中子流程的退出代码不感兴趣，您对他们从工作项中返回的价值感兴趣.这是因为在正常情况下，Pool中的进程要等到您close/join池后才会退出，因此在完成所有工作之前，没有要检索的退出代码，并且Pool大约是被摧毁.因此，没有公共API可以获取这些子流程的退出代码.

If you're using multiprocessing.Pool.map you're generally not interested in the exit code of the sub-processes in the pool, you're interested in what value they returned from their work item. This is because under normal conditions, the processes in a Pool won't exit until you close/join the pool, so there's no exit codes to retrieve until all work is complete, and the Pool is about to be destroyed. Because of this, there is no public API to get the exit codes of those sub-processes.

现在，您担心特殊情况，即带外某些东西在工作时会杀死其中一个子流程.如果遇到这样的问题，您可能会遇到一些奇怪的行为.实际上，在我的测试中，我在Pool中终止进程的同时正在执行map调用的一部分，因此map从未完成，因为终止的进程未完成.但是，Python确实立即启动了一个新进程来替换我杀死的进程.

Now, you're worried about exceptional conditions, where something out-of-band kills one of the sub-processes while it's doing work. If you hit an issue like this, you're probably going to run into some strange behavior. In fact, in my tests where I killed a process in a Pool while it was doing work as part of a map call, map never completed, because the killed process didn't complete. Python did, however, immediately launch a new process to replace the one I killed.

也就是说，您可以通过使用私有的_pool属性直接访问池中的multiprocessing.Process对象来获取池中每个进程的pid:

That said, you can get the pid of each process in your pool by accessing the multiprocessing.Process objects inside the pool directly, using the private _pool attribute:

pool = multiprocessing.Pool()
for proc in pool._pool:
  print proc.pid

因此，您可以尝试尝试检测进程何时意外终止(假设您不会因此而陷入阻塞调用中).您可以通过在调用map_async之前和之后检查池中的进程列表来做到这一点:

So, one thing you could do to try to detect when a process had died unexpectedly (assuming you don't get stuck in a blocking call as a result). You can do this by examining the list of processes in the pool before and after making a call to map_async:

before = pool._pool[:]  # Make a copy of the list of Process objects in our pool
result = pool.map_async(func, iterable)  # Use map_async so we don't get stuck.
while not result.ready():  # Wait for the call to complete
    if any(proc.exitcode for proc in before):  # Abort if one of our original processes is dead.
        print "One of our processes has exited. Something probably went horribly wrong."
        break
    result.wait(timeout=1)
else:  # We'll enter this block if we don't reach `break` above.
    print result.get() # Actually fetch the result list here.

我们必须复制列表，因为当Pool中的某个进程死亡时，Python会立即用一个新进程替换它，并从列表中删除死掉的进程.

We have to make a copy of the list because when a process in the Pool dies, Python immediately replaces it with a new process, and removes the dead one from the list.

这在我的测试中对我有用，但是因为它依赖于Pool对象(_pool)的私有属性，所以在生产代码中使用它是有风险的.我还建议，过多担心这种情况可能是矫kill过正，因为这种情况极不可能发生，并且会使实施变得非常复杂.

This worked for me in my tests, but because it's relying on a private attribute of the Pool object (_pool) it's risky to use in production code. I would also suggest that it may be overkill to worry too much about this scenario, since it's very unlikely to occur and complicates the implementation significantly.

这篇关于检索使用multiprocessing.Pool.map启动的进程的退出代码的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！