ipython 并行的 Python 名称空间问题 [英] Python name space issues with ipython parallel

问题描述

我开始尝试使用 IPython 并行工具，但遇到了一个问题.我使用以下命令启动我的 python 引擎:

ipcluster start -n 3

然后下面的代码运行良好:

from IPython.parallel import Clientdef dop(x):rc = 客户端()dview = rc[:]dview.block=真dview.execute('a = 5')dview['b'] = 10ack = dview.apply(lambda x: a+b+x, x)返回确认ack = dop(27)打印确认

按原样返回 [42, 42, 42].但是如果我将代码分解成不同的文件:dop.py:

from IPython.parallel import Clientdef dop(x):rc = 客户端()dview = rc[:]dview.block=真dview.execute('a = 5')dview['b'] = 10打印 dview['a']ack = dview.apply(lambda x: a+b+x, x)返回确认

并尝试以下操作:

from dop import dopack = dop(27)打印确认

我收到来自每个引擎的错误:

[0:apply]: NameError: 全局名称 'a' 未定义[1:apply]: NameError: 全局名称 'a' 未定义[2:apply]: NameError: 全局名称 'a' 未定义

我不明白...为什么我不能将函数放在不同的文件中并导入它?

解决方案

快速回答:使用 IPython.parallel.util[1] 中的 @interactive 装饰你的函数您希望它可以访问引擎的全局命名空间:

<前>从 IPython.parallel.util 导入交互f = 交互式(lambda x:a+b+x)ack = dview.apply(f, x)

实际解释:

IPython 用户命名空间本质上是模块 __main__.这是执行 execute('a = 5') 时运行代码的地方.

如果你交互定义一个函数，它的模块也是__main__:

<前>拉姆 = 拉姆达 x: a+b+xlam.__模块__'__主要的__'

当引擎反序列化一个函数时，它会在该函数模块的适当全局命名空间中进行，因此在您的客户端的 __main__ 中定义的函数也在 __main__ 中定义在引擎上，因此可以访问 a.

一旦你把它放在一个文件中并导入它，那么函数不再附加到__main__，而是模块dop:

<前>从 dop 导入 dopdop.__模块__'dop'

在该模块中常规定义的所有函数(包括 lambdas)都将具有此值，因此当它们在引擎上解包时，它们的全局命名空间将是 dop 模块的名称，不是 __main__，所以你的 'a' 是不可访问的.

出于这个原因，IPython 提供了一个简单的 @interactive 装饰器，它导致任何函数都被解包，就像它在 __main__ 中定义的一样，而不管函数实际在哪里定义.

举个例子，看看这个dop.py:

<前>从 IPython.parallel 导入客户端从 IPython.parallel.util 导入交互一 = 1def dop(x):rc = 客户端()dview = rc[:]dview['a'] = 5f = λ x: a+x返回 dview.apply_sync(f, x)定义 idop(x):rc = 客户端()dview = rc[:]dview['a'] = 5f = 交互式(lambda x:a+x)返回 dview.apply_sync(f, x)

现在，dop 将使用来自 dop 模块的 'a'，而 idop 将使用来自引擎命名空间的 'a'.两者的唯一区别是传递给 apply 的函数被包裹在 @interactive:

<前>从 dop 导入 dop，idop打印 dop(5) # 6打印 idop(5) # 10

[1]:在 IPython >= 0.13(即将发布)中，@interactive 也可以作为 from IPython.parallel import interactive 使用，它本来应该是.

I'm starting to experiment with the IPython parallel tools and have an issue. I start up my python engines with:

ipcluster start -n 3

Then the following code runs fine:

from IPython.parallel import Client

def dop(x):
    rc = Client()
    dview = rc[:]
    dview.block=True
    dview.execute('a = 5')
    dview['b'] = 10
    ack = dview.apply(lambda x: a+b+x, x)
    return ack

ack = dop(27)
print ack

returns [42, 42, 42] as it should. But if I break the code into different files: dop.py:

from IPython.parallel import Client

def dop(x):
    rc = Client()
    dview = rc[:]
    dview.block=True
    dview.execute('a = 5')
    dview['b'] = 10
    print dview['a']
    ack = dview.apply(lambda x: a+b+x, x)
    return ack

and try the following:

from dop import dop
ack = dop(27)
print ack

I get errors from each engine:

[0:apply]: NameError: global name 'a' is not defined
[1:apply]: NameError: global name 'a' is not defined
[2:apply]: NameError: global name 'a' is not defined

I don't get it...why can't I put the function in a different file and import it?

解决方案

Quick answer: decorate your function with @interactive from IPython.parallel.util[1] if you want it to have access to the engine's global namespace:

from IPython.parallel.util import interactive
f = interactive(lambda x: a+b+x)
ack = dview.apply(f, x)

The actual explanation:

the IPython user namespace is essentially the module __main__. This is where code is run when you do execute('a = 5').

If you define a function interactively, its module is also __main__:

lam = lambda x: a+b+x
lam.__module__
'__main__'

When the Engine unserializes a function, it does so in the appropriate global namespace for the function's module, so functions defined in __main__ in your client are also defined in __main__ on the Engine, and thus have access to a.

Once you put it in a file and import it, then the functions are no longer attached to __main__, but the module dop:

from dop import dop
dop.__module__
'dop'

All functions conventionally defined in that module (lambdas included) will have this value, so when they are unpacked on the Engine their global namespace will be that of the dop module, not __main__, so your 'a' is not accessible.

For this reason, IPython provides a simple @interactive decorator that results in any function being unpacked as if it were defined in __main__, regardless of where the function is actually defined.

For an example of the difference, take this dop.py:

from IPython.parallel import Client
from IPython.parallel.util import interactive

a = 1

def dop(x):
    rc = Client()
    dview = rc[:]
    dview['a'] = 5
    f = lambda x: a+x
    return dview.apply_sync(f, x)

def idop(x):
    rc = Client()
    dview = rc[:]
    dview['a'] = 5
    f = interactive(lambda x: a+x)
    return dview.apply_sync(f, x)

Now, dop will use 'a' from the dop module, and idop will use 'a' from your engine namespaces. The only difference between the two is that the function passed to apply is wrapped in @interactive:

from dop import dop, idop
print dop(5)  # 6
print idop(5) # 10

[1]: In IPython >= 0.13 (upcoming release), @interactive is also available as from IPython.parallel import interactive, where it always should have been.

这篇关于ipython 并行的 Python 名称空间问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！