使用sync_imports（）在IPython.parallel引擎上导入自定义模块 [英] Import custom modules on IPython.parallel engines with sync_imports()

问题描述

我一直在玩IPython.parallel，我想使用我自己的一些自定义模块，但是无法按照 dview.sync_imports（）的ipython-doc / stable / parallel / parallel_multiengine.htmlrel =noreferrer>食谱。对我来说唯一有用的东西就像

I've been playing around with IPython.parallel and I wanted to use some custom modules of my own, but haven't been able to do it as explained on the cookbook using dview.sync_imports(). The only thing that has worked for me was something like

def my_parallel_func(args):
    import sys
    sys.path.append('/path/to/my/module')
    import my_module
    #and all the rest

然后在主要的

if __name__=='__main__':
     #set up dview...
     dview.map( my_parallel_func, my_args )

在我看来，正确的做法是

The correct way to do this would in my opinion be something like

 with dview.sync_imports():
     import sys
     sys.path.append('/path/to/my/module')
     import my_module

但这会抛出错误，说没有名为 my_module的模块。

but this throws an error saying there is no module named my_module.

那么，使用 dview.sync_imports（） ??

推荐答案

问题是你要更改 PYTHONPATH 只是在运行客户端的本地进程中，而不是在 ipcluster 中运行的远程进程中。

The problem is that you're changing the PYTHONPATH just in the local process running the Client, and not in the remote processes running in the ipcluster.

如果你运行下一代代码，你可以观察到这种行为：

You can observe this behaviour if you run the next peace of code:

from IPython.parallel import Client

rc = Client()
dview = rc[:]

with dview.sync_imports():
    import sys
    sys.path[:] = ['something']

def parallel(x):
    import sys
    return sys.path

print 'Local: ', sys.path
print 'Remote: ', dview.map_sync(parallel, range(1))

基本上，您要与 sync_imports 一起使用的所有模块必须已经在 PYTHONPATH 中。

Basically all the modules that you want to use with sync_imports must already be in the PYTHONPATH.

如果它不在 PYTHONPATH 中，那么你必须将它添加到你执行remo的函数中的路径tely，然后在函数中导入模块。

If it's not in the PYTHONPATH then you must add it to the path in the function that you execute remotely, and then import the module in the function.

这篇关于使用sync_imports（）在IPython.parallel引擎上导入自定义模块的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！