从类定义中的列表推导式访问类变量 [英] Accessing class variables from a list comprehension in the class definition
问题描述
如何从类定义中的列表推导式访问其他类变量?以下内容在 Python 2 中有效，但在 Python 3 中失败:
类 Foo:x = 5y = [x for i in range(1)]Python 3.2 给出错误:
NameError: 全局名称 'x' 未定义尝试 Foo.x 也不起作用.关于如何在 Python 3 中执行此操作的任何想法?
一个稍微复杂的激励示例:
from collections import namedtuple类状态数据库:State = namedtuple('State', ['name', 'capital'])db = [State(*args) for args in [['阿拉巴马州'，'蒙哥马利']，['阿拉斯加'，'朱诺']，# ...]]在此示例中，apply() 本来是一个不错的解决方法，但遗憾的是它已从 Python 3 中删除.
类作用域和列表、集合或字典推导式以及生成器表达式不能混用.
为什么；或者，官方的说法
在 Python 3 中，列表推导式被赋予了自己的适当范围(本地命名空间)，以防止它们的本地变量渗入周围的范围(参见 即使在理解范围之后，列表理解也会重新绑定名称.这是对的吗?).在模块或函数中使用这样的列表推导式时，这很好，但在类中，范围界定有点，呃，奇怪.
这在 pep 227 中有记录:
<块引用>类范围内的名称不可访问.名称解析为最里面的封闭函数作用域.如果一个类定义发生在一系列嵌套作用域中，解析过程会跳过类定义.
以及class 复合语句文档:
然后在新的执行框架中执行该类的套件(参见 命名和绑定)，使用新创建的本地命名空间和原始全局命名空间.(通常，套件只包含函数定义.)当类的套件完成执行时，它的执行框架被丢弃，但它的本地命名空间被保存.[4] 然后使用继承列表为基类和为属性字典保存的本地命名空间.
强调我的；执行框架是临时作用域.
因为作用域被重新用作类对象的属性，允许它用作非局部作用域也会导致未定义的行为；例如，如果一个类方法将 x 作为嵌套作用域变量引用，然后操作 Foo.x 会发生什么?更重要的是，这对 Foo 的子类意味着什么?Python 必须区别对待类作用域，因为它与函数作用域非常不同.
最后但并非最不重要的是，链接的命名和绑定执行模型文档中的部分明确提到了类范围:
<块引用>类块中定义的名称范围仅限于类块；它没有扩展到方法的代码块——这包括推导式和生成器表达式，因为它们是使用函数作用域实现的.这意味着以下操作将失败:
A 类:一 = 42b = list(a + i for i in range(10))所以，总而言之:您不能从包含在该范围内的函数、列表推导式或生成器表达式访问类范围；它们就好像那个范围不存在一样.在 Python 2 中，列表推导式是使用快捷方式实现的，但在 Python 3 中，它们有自己的函数范围(因为它们一直应该有)，因此您的示例中断了.无论 Python 版本如何，其他理解类型都有自己的作用域，因此带有集合或字典理解的类似示例将在 Python 2 中中断.
# 同样的错误，在 Python 2 或 3 中y = {x: x for i in range(1)}(小)异常；或者，为什么一部分可能仍然有效
无论 Python 版本如何，都会在周围范围内执行推导式或生成器表达式的一部分.那将是最外层可迭代对象的表达式.在您的示例中，它是 range(1):
y = [x for i in range(1)]#^^^^^^^^^因此，在该表达式中使用 x 不会抛出错误:
# 运行良好y = [i for i in range(x)]这只适用于最外层的可迭代对象；如果一个推导式有多个 for 子句，则在推导式的范围内对内部 for 子句的可迭代对象进行评估:
# NameErrory = [i for i in range(1) for j in range(x)]#^^^^^^^^^^^^^^^^^ -----------------# 外循环 内循环，嵌套循环这个设计决定是为了在创建生成器表达式的最外层迭代时抛出错误而不是迭代时抛出错误，或者当最外层迭代结果不是迭代时.理解共享此行为以保持一致性.
在引擎盖下看；或者，比您想要的更多细节
您可以使用 dis 模块查看这一切.我在以下示例中使用 Python 3.3，因为它添加了 限定名称 整齐地标识我们要检查的代码对象.生成的字节码在功能上与 Python 3.2 相同.
为了创建一个类，Python 本质上采用了构成类主体的整个套件(因此所有内容都比 class 行更深一层缩进)，并像执行函数一样执行它:
第一个 LOAD_CONST 为 Foo 类主体加载一个代码对象，然后将它变成一个函数，并调用它.然后使用该调用的结果来创建类的名称空间，即它的 __dict__.到目前为止一切顺利.
这里需要注意的是，字节码包含一个嵌套的代码对象；在 Python 中，类定义、函数、推导式和生成器都表示为代码对象，其中不仅包含字节码，还包含表示局部变量、常量、从全局变量中获取的变量以及从嵌套作用域中获取的变量的结构.编译后的字节码引用这些结构，python 解释器知道如何访问这些给定的字节码.
这里要记住的重要一点是 Python 在编译时创建这些结构；class 套件是一个已经编译的代码对象("，第 2 行>).
让我们检查创建类体本身的代码对象；代码对象有一个 co_consts 结构:
上面的字节码创建了类主体.函数被执行，结果 locals() 命名空间，包含 x 和 y 用于创建类(除了它不工作，因为 x 没有定义为全局).注意在x中存储5后，会加载另一个code对象；这就是列表理解；它就像类体一样被包裹在一个函数对象中；创建的函数采用位置参数，range(1) 可迭代用于其循环代码，转换为迭代器.如字节码所示，range(1) 在类作用域内求值.
由此可以看出，函数或生成器的代码对象与推导式的代码对象之间的唯一区别是，后者在父代码对象立即执行被执行；字节码只是动态地创建一个函数并在几个小步骤中执行它.
Python 2.x 在那里使用内联字节码，这里是 Python 2.7 的输出:
2 0 LOAD_NAME 0 (__name__)3 STORE_NAME 1 (__module__)3 6 LOAD_CONST 0 (5)9 STORE_NAME 2 (x)4 12 BUILD_LIST 015 LOAD_NAME 3(范围)18 LOAD_CONST 1 (1)21 CALL_FUNCTION 124 GET_ITER>>25 FOR_ITER 12(到 40)28 STORE_NAME 4 (i)31 LOAD_NAME 2 (x)34 LIST_APPEND 237 绝对跳跃 25>>40 STORE_NAME 5 (y)第43话44 RETURN_VALUE没有加载代码对象，而是内联运行 FOR_ITER 循环.所以在 Python 3.x 中，列表生成器被赋予了自己的适当代码对象，这意味着它有自己的作用域.
然而，当模块或脚本首次被解释器加载时，推导式与 Python 源代码的其余部分一起编译，并且编译器不不将类套件视为有效范围.列表推导式中的任何引用变量都必须递归地查看包围类定义的范围.如果编译器未找到该变量，则将其标记为全局变量.列表推导代码对象的反汇编显示 x 确实是作为全局加载的:
这块字节码加载传入的第一个参数(range(1) 迭代器)，就像 Python 2.x 版本使用 FOR_ITER 循环它并创建它的输出.
如果我们在 foo 函数中定义了 x，x 将是一个单元格变量(单元格是指嵌套的作用域):<预><代码>>>>定义 foo():... x = 2... Foo 类:... x = 5... y = [x for i in range(1)]...返回Foo...>>>dis.dis(foo.__code__.co_consts[2].co_consts[2])5 0 BUILD_LIST 03 LOAD_FAST 0 (.0)>>6 FOR_ITER 12(到 21)9 STORE_FAST 1 (i)12 LOAD_DEREF 0 (x)15 LIST_APPEND 218 JUMP_ABSOLUTE 6>>21 RETURN_VALUE
LOAD_DEREF 将从代码对象单元格对象中间接加载 x :
实际引用从当前帧数据结构中查找值，这些数据结构是从函数对象的 .__closure__ 属性初始化的.由于为理解代码对象创建的函数再次被丢弃，我们无法检查该函数的闭包.要查看闭包的实际效果，我们必须改为检查嵌套函数:
总结一下:
- 列表推导式在 Python 3 中获得了自己的代码对象，对于函数、生成器或推导式的代码对象没有区别；理解代码对象被包装在一个临时函数对象中并立即调用.
- 代码对象在编译时创建，任何非局部变量都根据代码的嵌套范围被标记为全局变量或自由变量.类主体不被视为查找这些变量的范围.
- 在执行代码时，Python 只需查看全局变量，或当前正在执行的对象的闭包.由于编译器没有将类体作为作用域包含在内，因此不考虑临时函数命名空间.
解决方法；或者，怎么办
如果您要为 x 变量创建显式作用域，就像在函数中一样，您可以使用类作用域变量进行列表推导:
'temporary'y 函数可以直接调用；我们用它的返回值替换它.解析x时会考虑范围:
当然，阅读你的代码的人会对此有点头疼；你可能想在那里放一个很大的评论来解释你为什么这样做.
最好的解决方法是使用 __init__ 来创建一个实例变量:
def __init__(self):self.y = [self.x for i in range(1)]并避免所有令人头疼的问题，以及解释自己的问题.对于您自己的具体示例，我什至不会在类中存储 namedtuple；要么直接使用输出(根本不存储生成的类)，要么使用全局:
from collections import namedtupleState = namedtuple('State', ['name', 'capital'])类状态数据库:db = [State(*args) for args in [('阿拉巴马州'，'蒙哥马利')，('阿拉斯加', '朱诺'),# ...]]How do you access other class variables from a list comprehension within the class definition? The following works in Python 2 but fails in Python 3:
class Foo:
x = 5
y = [x for i in range(1)]
Python 3.2 gives the error:
NameError: global name 'x' is not defined
Trying Foo.x doesn't work either. Any ideas on how to do this in Python 3?
A slightly more complicated motivating example:
from collections import namedtuple
class StateDatabase:
State = namedtuple('State', ['name', 'capital'])
db = [State(*args) for args in [
['Alabama', 'Montgomery'],
['Alaska', 'Juneau'],
# ...
]]
In this example, apply() would have been a decent workaround, but it is sadly removed from Python 3.
Class scope and list, set or dictionary comprehensions, as well as generator expressions do not mix.
The why; or, the official word on this
In Python 3, list comprehensions were given a proper scope (local namespace) of their own, to prevent their local variables bleeding over into the surrounding scope (see List comprehension rebinds names even after scope of comprehension. Is this right?). That's great when using such a list comprehension in a module or in a function, but in classes, scoping is a little, uhm, strange.
This is documented in pep 227:
Names in class scope are not accessible. Names are resolved in the innermost enclosing function scope. If a class definition occurs in a chain of nested scopes, the resolution process skips class definitions.
and in the class compound statement documentation:
The class’s suite is then executed in a new execution frame (see section Naming and binding), using a newly created local namespace and the original global namespace. (Usually, the suite contains only function definitions.) When the class’s suite finishes execution, its execution frame is discarded but its local namespace is saved. [4] A class object is then created using the inheritance list for the base classes and the saved local namespace for the attribute dictionary.
Emphasis mine; the execution frame is the temporary scope.
Because the scope is repurposed as the attributes on a class object, allowing it to be used as a nonlocal scope as well leads to undefined behaviour; what would happen if a class method referred to x as a nested scope variable, then manipulates Foo.x as well, for example? More importantly, what would that mean for subclasses of Foo? Python has to treat a class scope differently as it is very different from a function scope.
Last, but definitely not least, the linked Naming and binding section in the Execution model documentation mentions class scopes explicitly:
The scope of names defined in a class block is limited to the class block; it does not extend to the code blocks of methods – this includes comprehensions and generator expressions since they are implemented using a function scope. This means that the following will fail:
class A: a = 42 b = list(a + i for i in range(10))
So, to summarize: you cannot access the class scope from functions, list comprehensions or generator expressions enclosed in that scope; they act as if that scope does not exist. In Python 2, list comprehensions were implemented using a shortcut, but in Python 3 they got their own function scope (as they should have had all along) and thus your example breaks. Other comprehension types have their own scope regardless of Python version, so a similar example with a set or dict comprehension would break in Python 2.
# Same error, in Python 2 or 3
y = {x: x for i in range(1)}
The (small) exception; or, why one part may still work
There's one part of a comprehension or generator expression that executes in the surrounding scope, regardless of Python version. That would be the expression for the outermost iterable. In your example, it's the range(1):
y = [x for i in range(1)]
# ^^^^^^^^
Thus, using x in that expression would not throw an error:
# Runs fine
y = [i for i in range(x)]
This only applies to the outermost iterable; if a comprehension has multiple for clauses, the iterables for inner for clauses are evaluated in the comprehension's scope:
# NameError
y = [i for i in range(1) for j in range(x)]
# ^^^^^^^^^^^^^^^^^ -----------------
# outer loop inner, nested loop
This design decision was made in order to throw an error at genexp creation time instead of iteration time when creating the outermost iterable of a generator expression throws an error, or when the outermost iterable turns out not to be iterable. Comprehensions share this behavior for consistency.
Looking under the hood; or, way more detail than you ever wanted
You can see this all in action using the dis module. I'm using Python 3.3 in the following examples, because it adds qualified names that neatly identify the code objects we want to inspect. The bytecode produced is otherwise functionally identical to Python 3.2.
To create a class, Python essentially takes the whole suite that makes up the class body (so everything indented one level deeper than the class <name>: line), and executes that as if it were a function:
>>> import dis
>>> def foo():
... class Foo:
... x = 5
... y = [x for i in range(1)]
... return Foo
...
>>> dis.dis(foo)
2 0 LOAD_BUILD_CLASS
1 LOAD_CONST 1 (<code object Foo at 0x10a436030, file "<stdin>", line 2>)
4 LOAD_CONST 2 ('Foo')
7 MAKE_FUNCTION 0
10 LOAD_CONST 2 ('Foo')
13 CALL_FUNCTION 2 (2 positional, 0 keyword pair)
16 STORE_FAST 0 (Foo)
5 19 LOAD_FAST 0 (Foo)
22 RETURN_VALUE
The first LOAD_CONST there loads a code object for the Foo class body, then makes that into a function, and calls it. The result of that call is then used to create the namespace of the class, its __dict__. So far so good.
The thing to note here is that the bytecode contains a nested code object; in Python, class definitions, functions, comprehensions and generators all are represented as code objects that contain not only bytecode, but also structures that represent local variables, constants, variables taken from globals, and variables taken from the nested scope. The compiled bytecode refers to those structures and the python interpreter knows how to access those given the bytecodes presented.
The important thing to remember here is that Python creates these structures at compile time; the class suite is a code object (<code object Foo at 0x10a436030, file "<stdin>", line 2>) that is already compiled.
Let's inspect that code object that creates the class body itself; code objects have a co_consts structure:
>>> foo.__code__.co_consts
(None, <code object Foo at 0x10a436030, file "<stdin>", line 2>, 'Foo')
>>> dis.dis(foo.__code__.co_consts[1])
2 0 LOAD_FAST 0 (__locals__)
3 STORE_LOCALS
4 LOAD_NAME 0 (__name__)
7 STORE_NAME 1 (__module__)
10 LOAD_CONST 0 ('foo.<locals>.Foo')
13 STORE_NAME 2 (__qualname__)
3 16 LOAD_CONST 1 (5)
19 STORE_NAME 3 (x)
4 22 LOAD_CONST 2 (<code object <listcomp> at 0x10a385420, file "<stdin>", line 4>)
25 LOAD_CONST 3 ('foo.<locals>.Foo.<listcomp>')
28 MAKE_FUNCTION 0
31 LOAD_NAME 4 (range)
34 LOAD_CONST 4 (1)
37 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
40 GET_ITER
41 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
44 STORE_NAME 5 (y)
47 LOAD_CONST 5 (None)
50 RETURN_VALUE
The above bytecode creates the class body. The function is executed and the resulting locals() namespace, containing x and y is used to create the class (except that it doesn't work because x isn't defined as a global). Note that after storing 5 in x, it loads another code object; that's the list comprehension; it is wrapped in a function object just like the class body was; the created function takes a positional argument, the range(1) iterable to use for its looping code, cast to an iterator. As shown in the bytecode, range(1) is evaluated in the class scope.
From this you can see that the only difference between a code object for a function or a generator, and a code object for a comprehension is that the latter is executed immediately when the parent code object is executed; the bytecode simply creates a function on the fly and executes it in a few small steps.
Python 2.x uses inline bytecode there instead, here is output from Python 2.7:
2 0 LOAD_NAME 0 (__name__)
3 STORE_NAME 1 (__module__)
3 6 LOAD_CONST 0 (5)
9 STORE_NAME 2 (x)
4 12 BUILD_LIST 0
15 LOAD_NAME 3 (range)
18 LOAD_CONST 1 (1)
21 CALL_FUNCTION 1
24 GET_ITER
>> 25 FOR_ITER 12 (to 40)
28 STORE_NAME 4 (i)
31 LOAD_NAME 2 (x)
34 LIST_APPEND 2
37 JUMP_ABSOLUTE 25
>> 40 STORE_NAME 5 (y)
43 LOAD_LOCALS
44 RETURN_VALUE
No code object is loaded, instead a FOR_ITER loop is run inline. So in Python 3.x, the list generator was given a proper code object of its own, which means it has its own scope.
However, the comprehension was compiled together with the rest of the python source code when the module or script was first loaded by the interpreter, and the compiler does not consider a class suite a valid scope. Any referenced variables in a list comprehension must look in the scope surrounding the class definition, recursively. If the variable wasn't found by the compiler, it marks it as a global. Disassembly of the list comprehension code object shows that x is indeed loaded as a global:
>>> foo.__code__.co_consts[1].co_consts
('foo.<locals>.Foo', 5, <code object <listcomp> at 0x10a385420, file "<stdin>", line 4>, 'foo.<locals>.Foo.<listcomp>', 1, None)
>>> dis.dis(foo.__code__.co_consts[1].co_consts[2])
4 0 BUILD_LIST 0
3 LOAD_FAST 0 (.0)
>> 6 FOR_ITER 12 (to 21)
9 STORE_FAST 1 (i)
12 LOAD_GLOBAL 0 (x)
15 LIST_APPEND 2
18 JUMP_ABSOLUTE 6
>> 21 RETURN_VALUE
This chunk of bytecode loads the first argument passed in (the range(1) iterator), and just like the Python 2.x version uses FOR_ITER to loop over it and create its output.
Had we defined x in the foo function instead, x would be a cell variable (cells refer to nested scopes):
>>> def foo():
... x = 2
... class Foo:
... x = 5
... y = [x for i in range(1)]
... return Foo
...
>>> dis.dis(foo.__code__.co_consts[2].co_consts[2])
5 0 BUILD_LIST 0
3 LOAD_FAST 0 (.0)
>> 6 FOR_ITER 12 (to 21)
9 STORE_FAST 1 (i)
12 LOAD_DEREF 0 (x)
15 LIST_APPEND 2
18 JUMP_ABSOLUTE 6
>> 21 RETURN_VALUE
The LOAD_DEREF will indirectly load x from the code object cell objects:
>>> foo.__code__.co_cellvars # foo function `x`
('x',)
>>> foo.__code__.co_consts[2].co_cellvars # Foo class, no cell variables
()
>>> foo.__code__.co_consts[2].co_consts[2].co_freevars # Refers to `x` in foo
('x',)
>>> foo().y
[2]
The actual referencing looks the value up from the current frame data structures, which were initialized from a function object's .__closure__ attribute. Since the function created for the comprehension code object is discarded again, we do not get to inspect that function's closure. To see a closure in action, we'd have to inspect a nested function instead:
>>> def spam(x):
... def eggs():
... return x
... return eggs
...
>>> spam(1).__code__.co_freevars
('x',)
>>> spam(1)()
1
>>> spam(1).__closure__
>>> spam(1).__closure__[0].cell_contents
1
>>> spam(5).__closure__[0].cell_contents
5
So, to summarize:
- List comprehensions get their own code objects in Python 3, and there is no difference between code objects for functions, generators or comprehensions; comprehension code objects are wrapped in a temporary function object and called immediately.
- Code objects are created at compile time, and any non-local variables are marked as either global or as free variables, based on the nested scopes of the code. The class body is not considered a scope for looking up those variables.
- When executing the code, Python has only to look into the globals, or the closure of the currently executing object. Since the compiler didn't include the class body as a scope, the temporary function namespace is not considered.
A workaround; or, what to do about it
If you were to create an explicit scope for the x variable, like in a function, you can use class-scope variables for a list comprehension:
>>> class Foo:
... x = 5
... def y(x):
... return [x for i in range(1)]
... y = y(x)
...
>>> Foo.y
[5]
The 'temporary' y function can be called directly; we replace it when we do with its return value. Its scope is considered when resolving x:
>>> foo.__code__.co_consts[1].co_consts[2]
<code object y at 0x10a5df5d0, file "<stdin>", line 4>
>>> foo.__code__.co_consts[1].co_consts[2].co_cellvars
('x',)
Of course, people reading your code will scratch their heads over this a little; you may want to put a big fat comment in there explaining why you are doing this.
The best work-around is to just use __init__ to create an instance variable instead:
def __init__(self):
self.y = [self.x for i in range(1)]
and avoid all the head-scratching, and questions to explain yourself. For your own concrete example, I would not even store the namedtuple on the class; either use the output directly (don't store the generated class at all), or use a global:
from collections import namedtuple
State = namedtuple('State', ['name', 'capital'])
class StateDatabase:
db = [State(*args) for args in [
('Alabama', 'Montgomery'),
('Alaska', 'Juneau'),
# ...
]]
这篇关于从类定义中的列表推导式访问类变量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
