多处理池示例不起作用并冻结内核 [英] multiprocessing pool example does not work and freeze the kernel

问题描述

我正在尝试并行化脚本，但是由于未知的原因，内核只是冻结而没有引发任何错误.

I'm trying to parallelize a script, but for an unknown reason the kernel just freeze without any errors thrown.

最小工作示例:

from multiprocessing import Pool

def f(x):
  return x*x

p = Pool(6)
print(p.map(f, range(10)))   

有趣的是，如果我在另一个文件中定义函数然后将其导入，则一切正常.我如何才能使它工作而无需另一个文件?

Interestingly, all works fine if I define my function in another file then import it. How can I make it work without the need of another file?

我使用spyder(anaconda)，如果从Windows命令行运行代码，则结果相同.

I work with spyder (anaconda) and I have the same result if I run my code from the windows command line.

推荐答案

之所以会发生这种情况，是因为在子进程导入f时，您没有保护代码的过程部分免于重新执行.

This happens because you didn't protect your procedural part of the code from re-execution when your child processes are importing f.

他们需要导入f，因为Windows不支持将分叉作为新进程的启动方法(仅生成).必须从头开始一个新的Python进程，并导入f，并且此导入还将触发在所有子进程中及其子进程及其子进程中创建另一个Pool.

They need to import f, because Windows doesn't support forking as start method for new processes (only spawn). A new Python process has to be started from scratch, f imported and this import will also trigger another Pool to be created in all child-processes ... and their child-processes and their child-processes.

为防止这种递归，您必须在上部(应在导入时运行)和下部(应仅在将脚本作为主脚本执行时运行)之间插入if __name__ == '__main__':-行给父母).

To prevent this recursion, you have to insert an if __name__ == '__main__': -line between the upper part, which should run on imports and a lower part, which should only run when your script is executed as the main script (only the case for the parent).

from multiprocessing import Pool

def f(x):
  return x*x

if __name__ == '__main__': # protect your program's entry point

    p = Pool(6)
    print(p.map(f, range(10))) 

在Windows和Unix-y系统上，如果与"forkserver"启动方法(而非默认的"fork")一起使用，则必须像这样分隔代码以进行多处理.

Separating your code like that is mandatory for multiprocessing on Windows and Unix-y systems when used with 'forkserver' start-method instead of default 'fork'.

通常的一个好习惯是，将" any "脚本分隔为较高的"definition"和较低的"execution as main"，以使代码可导入而无需不必要地执行部分，而仅当作为顶层脚本运行时才有意义.最后但并非最不重要的一点是，当您不混合定义和执行时，它有助于理解程序的控制流.

It's general a good practice to separate any script in upper "definition" and lower "execution as main", to make code importable without uncessessary executions of parts only relevant when run as top level script. Last but not least, it facilitates understanding the control flow of your program when you don't intermix definitions and executions.

这篇关于多处理池示例不起作用并冻结内核的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！