子类化 multiprocessing.managers.BaseProxy [英] Subclassing multiprocessing.managers.BaseProxy

问题描述

我在尝试实现新的 defaultdict 代理对象时遇到了一些麻烦.文档 有点吓人，所以我不知道如何正确地解决这个问题.

I'm having some trouble trying to implement a new defaultdict proxy object. The documentation is a bit scares, so I'm not sure how to go about this correctly.

我想将 defaultdict 添加到 Manager 实例可用的类型列表中.您不能在股票 multiprocessing.Manager 上使用 Manager.register 方法，所以我从 multiprocessing.mangers.BaseManager 制作了自己的存根管理器>

I want to add a defaultdict to the list of types that are available from the Manager instance. You cannot use the Manager.register method on the stock multiprocessing.Manager so I've made my own stub Manager from the multiprocessing.mangers.BaseManager

class Manager(BaseManager):
    pass

然后我创建了我的 multiprocessing.managers.BaseProxy 子类来容纳 defaultdict(我最初尝试使用花药存根，它可以子类化 defaultdictcode> 和 BaseProxy 但这似乎不起作用.这是我目前拥有的:

I then created my subclass of multiprocessing.managers.BaseProxy to house the defaultdict (I did initially try having anther stub which would subclass both defaultdict and BaseProxy but that didn't seem to work. Here's what I currently have:

class ProxyDefaultDict(BaseProxy):
    def __init__(self, default_factory=None, *args, **kwargs):
        self.__data = defaultdict(default_factory)
        super().__init__(*args, **kwargs)

    def _callmethod(self, methodname, args=(), kwds={}):
        return getattr(self.__data, methodname)(*args, **kwds)

    def _getvalue(self):
        return self.__data.copy()

    def __repr__(self):
        return self.__data.__repr__()

    def __str__(self):
        return self.__data.__str__()

Manager.register('defaultdict', ProxyDefaultDict)

最终目标是拥有一个共享字典，可以安全地跨进程和线程共享键锁.这是我如何对其进行初始化的示例:

The end goal is to have a shared dictionary which safely shares keyed Locks across processes and threads. Here's an example of how I image it would be initialised:

if __name__ == '__main__':
    m = Manager()
    d = m.defaultdict(m.Lock)
    with d['named_lock']:
        print('holding the lock')

但是，我遇到了一些问题:

However, I've hit a few problems:

1. BaseManager 的子类似乎只能通过上下文管理器初始化，即 with Manager() as m.在这种情况下，我将使用 m = Manager() - 正如 multiprocessing.Manager 允许的那样.不是世界末日，而是更好奇为什么会这样，如果这表明我做错了什么.

1. A subclass of BaseManager seems to be only initalizable via a context manager i.e. with Manager() as m. Which I would use m = Manager() in this case - as the multiprocessing.Manager allows. Not the end of the world but more curious why this is the case and if it's a sign I'm doing something incorrectly.

子类化 multiprocessing.managers.BaseManager 也意味着您会丢失 multiprocessing.Manager 中的默认注册值.在这种情况下，我需要为我的经理重新注册一个 ProxyLock(我也不确定这样做的预期方式).直接将 multiprocessing.Manager 子类化是否安全.

Subclassing multiprocessing.managers.BaseManager also menas you loose the default registered values from multiprocessing.Manager. In this case I need to re-register a ProxyLock for my manager (which I'm also unsure of the expected way to do this). Is it safe to just subclass multiprocessing.Manager directly.

最后，我的 ProxyDefaultDict 似乎不允许我干净地覆盖它的 __init__.而且我厌倦了在子类化时不调用 BaseProxy.__init__ .问题是 BaseProxy 也接受位置参数.我想解决这个问题的方法是使 default_factory 参数仅成为键控参数，但这会将预期的接口更改为 defaultdict 并让我假设我在这里做错了再次.Manager.Lock 等其他类型似乎能够接受位置参数.

Finally, my ProxyDefaultDict doesn't seem to allow my to cleanly override its __init__. And I'm weary of not calling the BaseProxy.__init__ when subclassing. The problem is that BaseProxy also accepts positional arguments. I guess the way round this is to make the default_factory argument a keyed argument only, but that changes the expected interface to defaultdict and makes me assume I'm doing something incorrectly here again. The other types like Manager.Lock seem to be able to accept positional arguments.

感谢您的帮助.

推荐答案

查看源代码后，对其稍作修改即可获得一个没有问题的 defaultdict 类型代理(基于内置 DictProxy 的创建方式).

After viewing the source code, a little modification of it works for me to get a defaultdict type proxy without issue (based on how the built in DictProxy is created).

from collections import defaultdict

from multiprocessing.managers import MakeProxyType, SyncManager

DefaultDictProxy = MakeProxyType("DefaultDictProxy", [
    '__contains__', '__delitem__', '__getitem__', '__len__',
    '__setitem__', 'clear', 'copy', 'default_factory', 'fromkeys',
    'get', 'items', 'keys', 'pop', 'popitem', 'setdefault',
    'update', 'values'])

SyncManager.register("defaultdict", defaultdict, DefaultDictProxy)
# Can also create your own Manager here, just using built in for simplicity

if __name__ == '__main__':
    with SyncManager() as sm:
        dd = sm.defaultdict(list)
        print(dd['a'])
        # []

我个人觉得使用已经提供的工具很方便，甚至不必担心如何自己子类化.

Personally I find it handy that by using the tools already provided, don't even need to worry about how to subclass it yourself.

但是，我认为这不会允许您创建您正在寻找的默认锁定方案.多处理锁被设计为只能被继承，并且通常锁不能被腌制，这是通过代理传输的数据类型的要求.示例:

However, I don't think that will allow you to create the default locks scenario you are looking for. Multiprocessing locks are designed to be inherited only, and in general Locks cannot be pickled, which is a requirement for data types being transferred through the proxies. Example:

    from multiprocessing import Lock

    m = SyncManager()
    m.start()
    d = m.defaultdict(Lock)
    print(d['named_lock'])
    m.shutdown()

会引发运行时错误:

RuntimeError: Lock objects should only be shared between processes through inheritance

这篇关于子类化 multiprocessing.managers.BaseProxy的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！