来自joblib的并行函数运行除函数外的整个代码 [英] Parallel function from joblib running whole code apart from functions

问题描述

我正在使用Python中 joblib 包中的 Parallel 函数.我只想将此功能用于处理我的其中一个功能，但不幸的是，整个代码(除了其他功能之外)都是并行运行的.

I am using Parallel function from joblib package in Python. I would like to use this function only for handle one of my functions but unfortunately the whole code is running in parallel (apart from other functions).

示例:

from joblib import Parallel, delayed
print ('I do not want this to be printed n times')
def do_something(arg):
    some calculations(arg)

Parallel(n_jobs=5)(delayed(do_something)(i) for i in range(0, n))

推荐答案

这是一个常见的错误，会错过文档中的设计指导.许多用户都遇到了同样的体验.

This is a common error to miss a design direction from documentation. Many users meet this very same piece of experience.

文档非常清楚，没有在 __main__ 引信之前放置def -s 之外的任何代码.

Documentation is quite clear about not placing any code but def-s before a __main__ fuse.

如果不这样做，错误确实会泛滥成灾，并给混乱带来沉重打击，但仍然有一个明确的建议可供您重新阅读文档，并无限泄漏到整个屏幕上:

If not doing so, errors indeed spray out and things turn wreck havoc, but still, an explicit advice to re-read the documentation is still present there, leaking infinitely over the screen:

[joblib] Attempting to do parallel computing
without protecting your import on a system that does not support forking.

To use parallel-computing in a script, you must protect your main loop
using "if __name__ == '__main__'".

Please see the joblib documentation on Parallel for more information

---

解决方案:

报道正确地完成了第一个问题融合的import保护，情况会变得更好:

---

Solution:

Having properly done the first issue, reported w.r.t. the fused import protection, things will get better:

C:\Python27.anaconda>python joblib_example.py
I do not want this to be printed n-times...
I do not want this to be printed n-times...
I do not want this to be printed n-times...
I do not want this to be printed n-times...
I do not want this to be printed n-times...
I do not want this to be printed n-times...

接下来的最后触摸，您就完成了:

next a final touch and you are done:

from sklearn.externals.joblib  import Parallel, delayed

def do_some_thing( arg ):
    pass
    return True

if  __name__ == '__main__': #################################### A __main__ FUSE:

    pass;                                   n = 6
    print "I do not want this to be printed n-times..."

    Parallel( n_jobs = 5 ) ( delayed( do_some_thing )( i )
                                                   for i in range( 0, n )
                             )

---

C:\Python27.anaconda>python joblib_example.py
I do not want this to be printed n-times...

C:\Python27.anaconda>

这篇关于来自joblib的并行函数运行除函数外的整个代码的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！