multiprocessing.jupyter笔记本中的池可在Linux上运行，但不能在Windows上运行 [英] multiprocessing.Pool in jupyter notebook works on linux but not windows

问题描述

我正在尝试运行一些独立的计算(尽管从相同的数据读取).我的代码在Ubuntu上运行时有效，但在Windows(Windows Server 2012 R2)上却无法运行，出现错误:

'module' object has no attribute ...

当我尝试使用multiprocessing.Pool时(它出现在内核控制台中，而不是笔记本本身的输出中)

(而且在创建池后定义函数的过程中，我已经犯了一个错误，并且我也已对其进行了更正，这不是问题.)

即使在最简单的示例中，也会发生这种情况:

from multiprocessing import Pool
def f(x):
    return x**2
pool = Pool(4)
for res in pool.map(f,range(20)):
    print res

我知道它需要能够导入模块(并且我不知道在笔记本电脑中工作时该模块如何工作)，而且我听说过IPython.Parallel，但是我找不到任何文档或示例.

任何解决方案/替代方案都将受到欢迎.

解决方案

由于我没有完整的答案，因此我会将其发布为评论，但我会进行修正，以便我进行了解.

>

from multiprocessing import Pool

def f(x):
    return x**2

if __name__ == '__main__':
    pool = Pool(4)
    for res in pool.map(f,range(20)):
        print(res)

这有效.我相信这个问题的答案是在这里.简而言之，子进程不知道它们是子进程，并且正在尝试递归运行主脚本.

这是我给出的错误，它为我们提供了相同的解决方案:

RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

I'm trying to run a few independent computations (though reading from the same data). My code works when I run it on Ubuntu, but not on Windows (windows server 2012 R2), where I get the error:

'module' object has no attribute ...

when I try to use multiprocessing.Pool (it appears in the kernel console, not as output in the notebook itself)

(And I've already made the mistake of defining the function AFTER creating the pool, and I've also corrected it, that's not the problem).

This happens even on the simplest of examples:

from multiprocessing import Pool
def f(x):
    return x**2
pool = Pool(4)
for res in pool.map(f,range(20)):
    print res

I know that it needs to be able to import the module (and I have no idea how this works when working in the notebook), and I've heard of IPython.Parallel, but I have been unable to find any documentation or examples.

Any solutions/alternatives would be most welcome.

解决方案

I would post this as a comment since I don't have a full answer, but I'll amend as I figure out what is going on.

from multiprocessing import Pool

def f(x):
    return x**2

if __name__ == '__main__':
    pool = Pool(4)
    for res in pool.map(f,range(20)):
        print(res)

This works. I believe the answer to this question is here. In short, the subprocesses do not know they are subprocesses and are attempting to run the main script recursively.

This is the error I am given, which gives us the same solution:

RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

这篇关于multiprocessing.jupyter笔记本中的池可在Linux上运行，但不能在Windows上运行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！