如何包装Python迭代器以使其线程安全？ [英] How to wrap a Python iterator to make it thread safe?

问题描述

因为有时它比围绕队列设计解决方案更实用，所以我想编写一个简单的包装器来使迭代器线程安全。到目前为止，我的灵感来自于这些 主题并提出两个想法：

Because sometimes it's more practical than designing a solution around queues, I would like to write a simple wrapper to make an iterator thread safe. So far, I had inspiration from these topics and came up with two ideas:

class LockedIterator(object):
    def __init__(self, it):
        self._lock = threading.Lock()
        self._it = it.__iter__()
        if hasattr(self._it, 'close'):
            def close(self):
                with self._lock:
                    self._it.close()
            self.__setattr__('close', close)

    def __iter__(self):
        return self

    def next(self):
        with self._lock:
            return self._it.next()

我不喜欢什么它，如果我必须指定所有可能的方法，它会变得有点冗长 - 好吧，我不能 - 例如发电机的特殊情况。此外，我可能有一些其他迭代器，甚至更具体的方法，现在已隐藏。

What I don't like about it, is that it gets a bit lengthy if I have to specify all possible methods - okay, I can't - such as the special case for generators. Also, I might have some other iterator with even more specific methods that have now become hidden.

class LockedIterator(object):
    def __init__(self, it):
        self._lock = threading.Lock()
        self._it = it.__iter__()

    def __getattr__(self, item):
        attr = getattr(self._it, item)
        if callable(attr):
            def hooked(*args, **kwargs):
                with self._lock:
                    return attr(*args, **kwargs)
            setattr(self, item, hooked)
            return hooked

这更简洁，但它只能拦截来电，而不是例如，直接的财产变化。 （现在隐藏这些属性以防止出现问题。）更重要的是，它使Python不再将我的对象识别为迭代器！

This is more concise, but it can only intercept calls, and not, for example, direct property changes. (Those properties are now hidden to prevent problems.) More importantly, it makes it so that Python does no longer recognize my object as an iterator!

什么是使所有迭代器（甚至更好：所有对象）都能工作的最佳方法，而不会产生漏洞抽象？我不是太担心锁定什么时候没有必要，但是如果你能想出一个解决这个问题的解决办法，那太好了！

What is the best way of making this work for all iterators (or even better: all objects), without creating a leaky abstraction? I'm not too worried about locking when it's not necessary, but if you can come up with a solution that circumvents that, great!

推荐答案

首先，您是否了解 GIL ？尝试编写多线程Python通常比使用简单的单线程版本的运行时间更慢。

First, are you aware of the GIL? Attempts to write multi-threaded Python typically end up in slower run-time than with a straightforward single-threaded version.

您首次尝试访问迭代器线程 - 安全似乎很合理。您可以使用生成器使其更具可读性：

Your first attempt at making access to an iterator thread-safe seems quite reasonable. You can make it a bit more readable by using a generator:

def locked_iter(it):
    it = iter(it)
    lock = threading.Lock()
    while True:
        try:
            with lock:
                value = next(it)
        except StopIteration:
            return
        yield value

这篇关于如何包装Python迭代器以使其线程安全？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！