Windows上的python joblib Parallel甚至不起作用"if __name__ =='__main__':"被添加 [英] python joblib Parallel on Windows not working even "if __name__ == '__main__':" is added

问题描述

我正在Windows上的Python中运行并行处理.这是我的代码:

I'm running parallel processing in Python on Windows. Here's my code:

from joblib import Parallel, delayed

def f(x): 
    return sqrt(x)

if __name__ == '__main__':
    a = Parallel(n_jobs=2)(delayed(f)(i) for i in range(10))

这是错误消息:

Process PoolWorker-2:  
Process PoolWorker-1:  
Traceback (most recent call last):    
File "C:\Users\yoyo__000.BIGBLACK\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.4.3105.win-x86_64\lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()   
File "C:\Users\yoyo__000.BIGBLACK\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.4.3105.win-x86_64\lib\multiprocessing\process.py", line 114, in run
self._target(*self._args, **self._kwargs)   
File "C:\Users\yoyo__000.BIGBLACK\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.4.3105.win-x86_64\lib\multiprocessing\pool.py", line 102, in worker
task = get()   
File "C:\Users\yoyo__000.BIGBLACK\AppData\Local\Enthought\Canopy\User\lib\site-packages\joblib\pool.py", line 363, in get
return recv()  
AttributeError: 'module' object has no attribute 'f'

推荐答案

根据此网站问题是Windows特定的:

According to this site the problem is Windows specific:

是的:在linux下，我们正在分叉，因此他们不需要腌制 功能，并且工作正常.在Windows下，该功能需要 可腌制，即需要从另一个文件中导入.这是 实际上是一种好习惯:将模块推入以进行重用.

Yes: under linux we are forking, thus their is no need to pickle the function, and it works fine. Under windows, the function needs to be pickleable, ie it needs to be imported from another file. This is actually good practice: making modules pushes for reuse.

我已经尝试过您的代码，并且在Linux下可以完美地工作. 在Windows下，如果它是通过脚本(如python script_with_your_code.py)运行的，则运行正常.但是在交互式python会话中运行时失败.当我将f函数保存在单独的模块中并将其导入到我的交互式会话中时，它为我工作.

I've tried your code and it works flawlessly under Linux. Under Windows it runs OK if it is run from a script, like python script_with_your_code.py. But it fails when ran in an interactive python session. It worked for me when I saved the f function in separate module and imported it into my interactive session.

不工作:
互动环节:

NOT WORKING:
Interactive session:

>>> from math import sqrt
>>> from joblib import Parallel, delayed

>>> def f(x):
...     return sqrt(x)

>>> if __name__ == '__main__':
...     a = Parallel(n_jobs=2)(delayed(f)(i) for i in range(10))
...
Process PoolWorker-1:
Traceback (most recent call last):
  File "C:\Python27\lib\multiprocessing\process.py", line 258, in _bootstrap
    self.run()
  File "C:\Python27\lib\multiprocessing\process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Python27\lib\multiprocessing\pool.py", line 102, in worker
    task = get()
  File "C:\Python27\lib\site-packages\joblib\pool.py", line 359, in get
    return recv()
AttributeError: 'module' object has no attribute 'f'

工作方式:
fun.py

WORKING:
fun.py

from math import sqrt

def f(x):
    return sqrt(x)

互动会话:

>>> from joblib import Parallel, delayed
>>> from fun import f

>>> if __name__ == '__main__':
...     a = Parallel(n_jobs=2)(delayed(f)(i) for i in range(10))
...
>>> a
[0.0, 1.0, 1.4142135623730951, 1.7320508075688772, 2.0, 2.23606797749979, 2.449489742783178, 2.6457513110645907, 2.8284271247461903, 3.0]

这篇关于Windows上的python joblib Parallel甚至不起作用"if __name__ =='__main__':"被添加的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！