列表理解和生成器表达式中的收益 [英] yield in list comprehensions and generator expressions
问题描述
以下行为对我来说似乎很违反直觉(Python 3.4):
The following behaviour seems rather counterintuitive to me (Python 3.4):
>>> [(yield i) for i in range(3)]
<generator object <listcomp> at 0x0245C148>
>>> list([(yield i) for i in range(3)])
[0, 1, 2]
>>> list((yield i) for i in range(3))
[0, None, 1, None, 2, None]
最后一行的中间值实际上并不总是None，它们是我们生成器中的send，等于(我想)以下生成器:
The intermediate values of the last line are actually not always None, they are whatever we send into the generator, equivalent (I guess) to the following generator:
def f():
for i in range(3):
yield (yield i)
这三行代码完全起作用，这让我感到很有趣. 参考说，yield仅在函数定义中允许(尽管我可以读取错误和/或它可能只是从旧版本复制而来).前两行在Python 2.7中生成SyntaxError，但第三行则不.
It strikes me as funny that those three lines work at all. The Reference says that yield is only allowed in a function definition (though I may be reading it wrong and/or it may simply have been copied from the older version). The first two lines produce a SyntaxError in Python 2.7, but the third line doesn't.
而且，这看起来很奇怪
- 列表理解会返回生成器而不是列表
- 并且生成器表达式转换为列表，并且相应的列表推导包含不同的值.
有人可以提供更多信息吗?
Could someone provide more information?
推荐答案
注意:这是CPython在理解和生成器表达式中对
yield的处理中的错误，已在Python 3.8中修复，并在Python 3.7中弃用了警告.请参阅 Python错误报告和
Note: this was a bug in the CPython's handling of
yieldin comprehensions and generator expressions, fixed in Python 3.8, with a deprecation warning in Python 3.7. See the Python bug report and the What's New entries for Python 3.7 and Python 3.8.
生成器表达式以及set和dict理解将编译为(生成器)函数对象.在Python 3中，列表推导得到相同的处理.从本质上讲，它们都是一个新的嵌套范围.
Generator expressions, and set and dict comprehensions are compiled to (generator) function objects. In Python 3, list comprehensions get the same treatment; they are all, in essence, a new nested scope.
如果尝试反汇编生成器表达式，则可以看到以下内容:
You can see this if you try to disassemble a generator expression:
>>> dis.dis(compile("(i for i in range(3))", '', 'exec'))
1 0 LOAD_CONST 0 (<code object <genexpr> at 0x10f7530c0, file "", line 1>)
3 LOAD_CONST 1 ('<genexpr>')
6 MAKE_FUNCTION 0
9 LOAD_NAME 0 (range)
12 LOAD_CONST 2 (3)
15 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
18 GET_ITER
19 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
22 POP_TOP
23 LOAD_CONST 3 (None)
26 RETURN_VALUE
>>> dis.dis(compile("(i for i in range(3))", '', 'exec').co_consts[0])
1 0 LOAD_FAST 0 (.0)
>> 3 FOR_ITER 11 (to 17)
6 STORE_FAST 1 (i)
9 LOAD_FAST 1 (i)
12 YIELD_VALUE
13 POP_TOP
14 JUMP_ABSOLUTE 3
>> 17 LOAD_CONST 0 (None)
20 RETURN_VALUE
上面显示了生成器表达式被编译为代码对象，并作为函数加载(MAKE_FUNCTION从代码对象创建函数对象). .co_consts[0]参考使我们可以看到为表达式生成的代码对象，并且它使用YIELD_VALUE就像生成器函数一样.
The above shows that a generator expression is compiled to a code object, loaded as a function (MAKE_FUNCTION creates the function object from the code object). The .co_consts[0] reference lets us see the code object generated for the expression, and it uses YIELD_VALUE just like a generator function would.
因此，yield表达式在该上下文中起作用，因为编译器将它们视为伪装函数.
As such, the yield expression works in that context, as the compiler sees these as functions-in-disguise.
这是一个错误； yield在这些表达式中没有位置. Python 3.7之前的Python 语法允许这样做(这就是为什么代码可编译的原因)，但是
This is a bug; yield has no place in these expressions. The Python grammar before Python 3.7 allows it (which is why the code is compilable), but the yield expression specification shows that using yield here should not actually work:
yield表达式仅在定义 generator 函数时使用，因此只能在函数定义的主体中使用.
The yield expression is only used when defining a generator function and thus can only be used in the body of a function definition.
已确认这是 issue 10544 中的错误.该错误的解决方案是使用yield和yield from将它会引发DeprecationWarning 以确保代码停止使用此代码构造.如果使用 -3命令行开关启用Python 3兼容性警告.
This has been confirmed to be a bug in issue 10544. The resolution of the bug is that using yield and yield from will raise a SyntaxError in Python 3.8; in Python 3.7 it raises a DeprecationWarning to ensure code stops using this construct. You'll see the same warning in Python 2.7.15 and up if you use the -3 command line switch enabling Python 3 compatibility warnings.
3.7.0b1警告看起来像这样；将警告变成错误会给您一个SyntaxError异常，就像在3.8中一样:
The 3.7.0b1 warning looks like this; turning warnings into errors gives you a SyntaxError exception, like you would in 3.8:
>>> [(yield i) for i in range(3)]
<stdin>:1: DeprecationWarning: 'yield' inside list comprehension
<generator object <listcomp> at 0x1092ec7c8>
>>> import warnings
>>> warnings.simplefilter('error')
>>> [(yield i) for i in range(3)]
File "<stdin>", line 1
SyntaxError: 'yield' inside list comprehension
列表理解中的yield和生成器表达式中的yield的操作方式之间的差异源于这两个表达式的实现方式上的差异.在Python 3中，列表推导使用LIST_APPEND调用将堆栈顶部添加到要构建的列表中，而生成器表达式则产生该值.添加(yield <expr>)只会向其中一个添加另一个YIELD_VALUE操作码:
The differences between how yield in a list comprehension and yield in a generator expression operate stem from the differences in how these two expressions are implemented. In Python 3 a list comprehension uses LIST_APPEND calls to add the top of the stack to the list being built, while a generator expression instead yields that value. Adding in (yield <expr>) just adds another YIELD_VALUE opcode to either:
>>> dis.dis(compile("[(yield i) for i in range(3)]", '', 'exec').co_consts[0])
1 0 BUILD_LIST 0
3 LOAD_FAST 0 (.0)
>> 6 FOR_ITER 13 (to 22)
9 STORE_FAST 1 (i)
12 LOAD_FAST 1 (i)
15 YIELD_VALUE
16 LIST_APPEND 2
19 JUMP_ABSOLUTE 6
>> 22 RETURN_VALUE
>>> dis.dis(compile("((yield i) for i in range(3))", '', 'exec').co_consts[0])
1 0 LOAD_FAST 0 (.0)
>> 3 FOR_ITER 12 (to 18)
6 STORE_FAST 1 (i)
9 LOAD_FAST 1 (i)
12 YIELD_VALUE
13 YIELD_VALUE
14 POP_TOP
15 JUMP_ABSOLUTE 3
>> 18 LOAD_CONST 0 (None)
21 RETURN_VALUE
分别位于字节码索引15和12的YIELD_VALUE操作码是多余的，是巢中的杜鹃.因此，对于list-comprehension-turned-generator，您每次生成1都会产生堆栈的顶部(使用yield返回值替换堆栈的顶部)，而对于Generator表达式变体，您将产生顶部的收益.堆栈(整数)，然后再次产生 ，但是现在堆栈包含yield的返回值，您第二次得到None.
The YIELD_VALUE opcode at bytecode indexes 15 and 12 respectively is extra, a cuckoo in the nest. So for the list-comprehension-turned-generator you have 1 yield producing the top of the stack each time (replacing the top of the stack with the yield return value), and for the generator expression variant you yield the top of the stack (the integer) and then yield again, but now the stack contains the return value of the yield and you get None that second time.
然后，对于列表理解，仍将返回预期的list对象输出，但是Python 3将其视为生成器，因此将返回值附加到
For the list comprehension then, the intended list object output is still returned, but Python 3 sees this as a generator so the return value is instead attached to the StopIteration exception as the value attribute:
>>> from itertools import islice
>>> listgen = [(yield i) for i in range(3)]
>>> list(islice(listgen, 3)) # avoid exhausting the generator
[0, 1, 2]
>>> try:
... next(listgen)
... except StopIteration as si:
... print(si.value)
...
[None, None, None]
那些None对象是yield表达式的返回值.
Those None objects are the return values from the yield expressions.
再次重申这一点；同样的问题也适用于Python 2和Python 3中的字典和集合理解；在Python 2中，yield返回值仍被添加到预期的字典或set对象中，并且返回值最后一次被屈服"，而不是附加到StopIteration异常:
And to reiterate this again; this same issue applies to dictionary and set comprehension in Python 2 and Python 3 as well; in Python 2 the yield return values are still added to the intended dictionary or set object, and the return value is 'yielded' last instead of attached to the StopIteration exception:
>>> list({(yield k): (yield v) for k, v in {'foo': 'bar', 'spam': 'eggs'}.items()})
['bar', 'foo', 'eggs', 'spam', {None: None}]
>>> list({(yield i) for i in range(3)})
[0, 1, 2, set([None])]
这篇关于列表理解和生成器表达式中的收益的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
