Python 3.4多处理递归Pool.map() [英] Python 3.4 multiprocessing recursive Pool.map()

问题描述

我正在Ubuntu 14.04上使用Python 3.4进行开发.我试图进行递归Pool.map().调用g()后，它会挂在那里并且永远不会返回.

I'm developing with Python 3.4 on Ubuntu 14.04. I was trying to do recursive Pool.map(). After I invoke g(), it hangs there and never returns.

import multiprocessing as mp

pool = mp.Pool()

def d(x):
    return x / 2.0


def f(x):
    w = pool.map(d, x)
    return w

def g():
    v = pool.map(f, [[1, 2], [3, 4]])

    print(v)

推荐答案

这是不可能的. Pool对象本身不能在进程之间安全地共享，因此不能同时在f和g中使用同一池.即使您可以执行此操作，您也会很快导致挂起，因为您的池仅限于cpu_count()个并发工作程序.一旦开始递归地创建更多的工作程序，您最终将拥有超过cpu_count()个工作程序，这将永远无法完成.正在运行的工作人员将等待池中排队的任务，但是排队的任务将永远无法启动，因为正在运行的任务正在等待.因此，您最终陷入僵局.简而言之:不要尝试这样做.

This isn't possible. The Pool object itself can't safely be shared between processes, so the same pool can't be used in both f and g. Even if you could do this, you'd quickly cause a hang, because your pool is limited to cpu_count() concurrent workers. Once you start creating more workers recursively, you'll end up with more than cpu_count() workers, which will never be able to finish; the running workers would be waiting on tasks that are queued up in the pool, but the queued up tasks won't ever be able to start because the running tasks are waiting. So you end up deadlocked. In short: don't try to do this.

这篇关于Python 3.4多处理递归Pool.map()的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！