如何创建正确收集垃圾的自定义生成器类 [英] How to create a custom generator class that is correctly garbage collected

问题描述

我正在尝试用Python编写一个充当生成器对象的类，尤其是在其被垃圾回收时调用了.close()的类.这很重要，因为这意味着当生成器被中断时，我可以确保将其清理干净，例如关闭文件或释放锁.

I'm trying to write a class in Python that behaves as a generator object, particularly in that when it's garbage collected .close() is called on it. That's important because it means that when the generator is interrupted I can make sure it'll clean up after itself, for example closing files or releasing locks.

以下是一些解释性代码: 如果您中断了一个生成器，那么当它被垃圾回收时，Python会在生成器对象上调用.close()，这会在生成器中抛出一个GeneratorExit错误，可以捕获该错误以进行清理，如下所示:

Here's some explanatory code: If you interupt a generator, then when it's garbage collected, Python calls .close() on the generator object, which throws a GeneratorExit error into the generator, which can be caught to allow cleanup, like follows:

from threading import Lock

lock = Lock()

def CustomGenerator(n, lock):
    lock.acquire()
    print("Generator Started: I grabbed a lock")
    try:
        for i in range(n):
            yield i
    except GeneratorExit:
        lock.release()
        print("Generator exited early: I let go of the lock")
        raise
    print("Generator finished successfully: I let go of the lock")

for i in CustomGenerator(100, lock):
    print("Received ", i)
    time.sleep(0.02)
    if i==3:
        break

if not lock.acquire(blocking=False):
    print("Oops: Finished, but lock wasn't released")
else:
    print("Finished: Lock was free")
    lock.release()

Generator Started: I grabbed a lock
Received  0
Received  1
Received  2
Received  3
Generator exited early: I let go of the lock
Finished: Lock was free

但是，如果您尝试通过继承collections.abc.Generator来实现自己的生成器对象，Python似乎并没有注意到在收集对象时应该调用close:

However, if you try to implement your own generator object by inheriting from collections.abc.Generator, Python doesn't seem to notice that it should call close when the object is collected:

from collections.abc import Generator
class CustomGeneratorClass(Generator):
    def __init__(self, n, lock):
        super().__init__()
        self.lock = lock
        self.lock.acquire()
        print("Generator Class Initialised: I grabbed a lock")
        self.n = n
        self.c = 0

    def send(self, arg):
        value = self.c
        if value >= self.n:
            raise StopIteration
        self.c += 1
        return value

    def throw(self, type, value=None, traceback=None):
        print("Exception Thrown in Generator: I let go of the lock")
        self.lock.release()
        raise StopIteration

for i in CustomGeneratorClass(100, lock):
    print("Received ", i)
    time.sleep(0.02)
    if i==3:
        break

if not lock.acquire(blocking=False):
    print("Oops: Finished, but lock wasn't released")
else:
    print("Finished: Lock was free")
    lock.release()

Generator Class Initialised: I grabbed a lock
Received  0
Received  1
Received  2
Received  3
Oops: Finished, but lock wasn't released

我认为继承Generator足以说服python我的CustomGeneratorClass是一个生成器，并且在垃圾回收时应该调用.close().

I thought that inheriting Generator would be sufficient to convince python that my CustomGeneratorClass was a generator and should have .close() called on it when garbage collected.

我认为这与以下事实有关:虽然生成器对象"是某种特殊的Generator:

I assume this has something to do with the fact that while 'generator object' are some kind of special Generator:

from types import GeneratorType

c_gen = CustomGenerator(100)
c_gen_class = CustomGeneratorClass(100)

print("CustomGenerator is a Generator:", isinstance(c_gen, Generator))
print("CustomGenerator is a GeneratorType:",isinstance(c_gen, GeneratorType))

print("CustomGeneratorClass is a Generator:",isinstance(c_gen_class, Generator))
print("CustomGeneratorClass is a GeneratorType:",isinstance(c_gen_class, GeneratorType))

CustomGenerator is a Generator: True
CustomGenerator is a GeneratorType: True
CustomGeneratorClass is a Generator: True
CustomGeneratorClass is a GeneratorType: False

我可以使用户定义的类对象为GeneratorType吗?

Can I make a user defined class object that is GeneratorType?

关于python如何决定调用.close()的事情，我是否不了解?

Is there something I don't understand about how python decides what to call .close() on?

如何确保在自定义生成器上调用.close()?

How can I ensure that .close() is called on my custom generator?

此问题不是如何编写生成器类的重复项. 对于实际制作生成器类，该问题的可接受答案确实建议了我在这里尝试的结构，它是生成器类，但未正确收集垃圾，如上面的代码所示.

This question is not a duplicate of How to write a generator class. For actually making a generator class, the accepted answer for that question does recommends exactly the structure I'm trying here, which is a generator class but is not correctly garbage collected, as shown in the code above.

推荐答案

PEP342 ，状态:

[generator].__del__()是[generator].close()的包装.当生成器对象被垃圾回收时将调用此方法...

[generator].__del__() is a wrapper for [generator].close(). This will be called when the generator object is garbage-collected ...

collections.abc 中的Generator类不实现__del__，也不实现其超类或元类.

The Generator class in collections.abc does not implement __del__, and neither do its superclasses or metaclass.

在问题中的类中添加__del__的这种实现会导致释放锁:

Adding this implementation of __del__ to the class in the question results in the lock being freed:

class CustomGeneratorClass(Generator):

    ...

    def __del__(self):
        self.close() 

输出:

Generator Class Initialised: I grabbed a lock
Recieved  0
Recieved  1
Recieved  2
Recieved  3
Exception Thrown in Generator: I let go of the lock
Finished: Lock was free

注意:

我对Python复杂的对象完成没有经验，因此应认真审查此建议，并测试其破坏性.特别是语言中有关__del__的警告.参考.

I'm not experienced with the intricacies of object finalisation in Python, so this suggestion should be examined critically, and tested to destruction. In particular, the warnings about __del__ in the language reference should be considered.

更高级的解决方案是在上下文管理器中运行生成器

A higher-level solution would be to run the generator in a context manager

with contextlib.closing(CustomGeneratorClass(100, lock)):
    # do stuff

但这很麻烦，并且依赖于记住这一点的代码用户.

but this is cumbersome, and relies on users of the code remembering to do it.

这篇关于如何创建正确收集垃圾的自定义生成器类的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！