如何在生成器中使用python上下文管理器 [英] How to use a python context manager inside a generator

问题描述

在python中，应在生成器内部使用with语句吗?需要明确的是，我并不是想使用装饰器通过生成器函数创建上下文管理器.我问在生成器内部使用with语句作为上下文管理器是否存在固有的问题，因为它至少在某些情况下会捕获StopIteration和GeneratorExit异常.以下是两个示例.

In python, should with-statements be used inside a generator? To be clear, I am not asking about using a decorator to create a context manager from a generator function. I am asking whether there is an inherent issue using a with-statement as a context manager inside a generator as it will catch StopIteration and GeneratorExit exceptions in at least some cases. Two examples follow.

比兹利的例子(第106页)提出了一个很好的例子.我已经对其进行了修改，以使用with语句，以便在opener方法中的yield之后显式关闭文件.我还添加了两种方法，可以在迭代结果时引发异常.

A good example of the issue is raised by Beazley's example (page 106). I have modified it to use a with statement so that the files are explicitly closed after the yield in the opener method. I have also added two ways that an exception can be thrown while iterating the results.

import os
import fnmatch

def find_files(topdir, pattern):
    for path, dirname, filelist in os.walk(topdir):
        for name in filelist:
            if fnmatch.fnmatch(name, pattern):
                yield os.path.join(path,name)
def opener(filenames):
    f = None
    for name in filenames:
        print "F before open: '%s'" % f
        #f = open(name,'r')
        with open(name,'r') as f:
            print "Fname: %s, F#: %d" % (name, f.fileno())
            yield f
            print "F after yield: '%s'" % f
def cat(filelist):
    for i,f in enumerate(filelist):
        if i ==20:
            # Cause and exception
            f.write('foobar')
        for line in f:
            yield line
def grep(pattern,lines):
    for line in lines:
        if pattern in line:
            yield line

pylogs = find_files("/var/log","*.log*")
files = opener(pylogs)
lines = cat(files)
pylines = grep("python", lines)
i = 0
for line in pylines:
    i +=1
    if i == 10:
        raise RuntimeError("You're hosed!")

print 'Counted %d lines\n' % i

在此示例中，上下文管理器成功关闭了打开器功能中的文件.当引发异常时，我看到了从异常追溯的轨迹，但是生成器无声地停止了.如果with语句捕获到异常，为什么生成器不继续?

In this example, the context manager successfully closes the files in the opener function. When an exception is raised, I see the trace back from the exception, but the generator stops silently. If the with-statement catches the exception why doesn't the generator continue?

当我定义自己的上下文管理器以在生成器中使用时.我收到运行时错误消息，说我已经忽略了GeneratorExit.例如:

When I define my own context managers for use inside a generator. I get runtime errors saying that I have ignored a GeneratorExit. For example:

class CManager(object):  
    def __enter__(self):
          print "  __enter__"
          return self
    def __exit__(self, exctype, value, tb):
        print "  __exit__; excptype: '%s'; value: '%s'" % (exctype, value)
        return True

def foo(n):
    for i in xrange(n):
        with CManager() as cman:
            cman.val = i
            yield cman
# Case1 
for item in foo(10):
    print 'Pass - val: %d' % item.val
# Case2
for item in foo(10):
    print 'Fail - val: %d' % item.val
    item.not_an_attribute

这个小演示在case1中工作正常，没有引发任何异常，但是在case2中出现了属性错误时失败.在这里，我看到了RuntimeException的出现，因为with语句捕获并忽略了GeneratorExit异常.

This little demo works fine in case1 with no exceptions raised, but fails in case2 where an attribute error is raised. Here I see a RuntimeException raised because the with statement has caught and ignored a GeneratorExit exception.

有人可以帮助澄清这个棘手的用例的规则吗?我怀疑这是我在做的事情，还是在我的__exit__方法中没有做的事情.我尝试添加代码以重新引发GeneratorExit，但这无济于事.

Can someone help clarify the rules for this tricky use case? I suspect it is something I am doing, or not doing in my __exit__ method. I tried adding code to re-raise GeneratorExit, but that did not help.

推荐答案

对于object.__exit__

如果提供了异常，并且该方法希望抑制该异常(即防止其传播)，则它应返回一个真值.否则，退出此方法后，异常将被正常处理.

If an exception is supplied, and the method wishes to suppress the exception (i.e., prevent it from being propagated), it should return a true value. Otherwise, the exception will be processed normally upon exit from this method.

在您的__exit__函数中，您将返回True，这将禁止所有 异常.如果将其更改为返回False，则异常将继续按正常方式引发(唯一的区别是，您保证调用了__exit__函数，并且可以确保自己执行清除操作)

In your __exit__ function, you're returning True which will suppress all exceptions. If you change it to return False, the exceptions will continue to be raised as normal (with the only difference being that you guarantee that your __exit__ function gets called and you can make sure to clean up after yourself)

例如，将代码更改为:

def __exit__(self, exctype, value, tb):
    print "  __exit__; excptype: '%s'; value: '%s'" % (exctype, value)
    if exctype is GeneratorExit:
        return False
    return True

允许您做正确的事情，而不压制GeneratorExit.现在，您仅看到属性错误.也许经验法则应该与任何异常处理相同-仅在知道如何处理异常的情况下才拦截异常. __exit__返回值True与裸露相同(也许稍微更糟！)，除了:

allows you to do the right thing and not suppress the GeneratorExit. Now you only see the attribute error. Maybe the rule of thumb should be the same as with any Exception handling -- only intercept Exceptions if you know how to handle them. Having an __exit__ return True is on par (maybe slightly worse!) than having a bare except:

try:
   something()
except: #Uh-Oh
   pass

---

请注意，当AttributeError升高(但未被捕获)时，我相信这会导致生成器对象上的引用计数下降到0，这将在生成器内触发GeneratorExit异常，以便它可以清除自身向上.使用我的__exit__，处理以下两种情况，希望您会明白我的意思:

---

Note that when the AttributeError is raised (and not caught), I believe that causes the reference count on your generator object to drop to 0 which then triggers a GeneratorExit exception within the generator so that it can clean itself up. Using my __exit__, play around with the following two cases and hopefully you'll see what I mean:

try:
    for item in foo(10):
        print 'Fail - val: %d' % item.val
        item.not_an_attribute
except AttributeError:
    pass

print "Here"  #No reference to the generator left.  
              #Should see __exit__ before "Here"

和

g = foo(10)
try:
    for item in g:
        print 'Fail - val: %d' % item.val
        item.not_an_attribute
except AttributeError:
    pass

print "Here"
b = g  #keep a reference to prevent the reference counter from cleaning this up.
       #Now we see __exit__ *after* "Here"

这篇关于如何在生成器中使用python上下文管理器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！