如何从多处理队列继承? [英] How to inherit from a multiprocessing queue?

问题描述

使用以下代码，似乎未初始化传递给工作程序的队列实例:

With the following code, it seems that the queue instance passed to the worker isn't initialized:

from multiprocessing import Process
from multiprocessing.queues import Queue

class MyQueue(Queue):

    def __init__(self, name):
        Queue.__init__(self)
        self.name = name

def worker(queue):
    print queue.name

if __name__ == "__main__":
    queue = MyQueue("My Queue")
    p = Process(target=worker, args=(queue,))
    p.start()
    p.join()

抛出:

... line 14, in worker
    print queue.name
AttributeError: 'MyQueue' object has no attribute 'name'

我无法重新初始化队列，因为我将失去queue.name的原始值，甚至将队列的名称作为参数传递给worker(这应该起作用，但这不是一个干净的解决方案).

I can't re-initialize the queue, because I'll loose the original value of queue.name, even passing the queue's name as an argument to the worker (this should work, but it's not a clean solution).

那么，如何从multiprocessing.queues.Queue继承而不会出现此错误?

So, how can inherit from multiprocessing.queues.Queue without getting this error?

推荐答案

在POSIX上，Queue对象通过简单继承被共享给子进程.*

On POSIX, Queue objects are shared to the child processes by simple inheritance.*

在Windows上是不可能的，因此它必须腌制Queue，通过管道将其发送给孩子，然后解开它.

On Windows, that isn't possible, so it has to pickle the Queue, send it over a pipe to the child, and unpickle it.

(这可能不是很明显，因为如果您实际上尝试腌制Queue，则会得到一个异常，RuntimeError: MyQueue objects should only be shared between processes through inheritance.如果您仔细查看源代码，您会发现这确实是一个谎言-它是谎言.仅当您尝试在multiprocess不在生成子进程的中间时腌制Queue时，才会引发此异常.)

(This may not be obvious, because if you actually try to pickle a Queue, you get an exception, RuntimeError: MyQueue objects should only be shared between processes through inheritance. If you look through the source, you'll see that this is really a lie—it only raises this exception if you try to pickle a Queue when multiprocess is not in the middle of spawning a child process.)

当然，一般的酸洗和酸洗都没有用，因为您最终将拥有两个相同的队列，而不是两个进程中的相同队列.因此，multiprocessing通过添加register_after_fork机制使对象在进行酸洗时使用，从而扩展了内容.**如果您查看

Of course generic pickling and unpickling wouldn't do any good, because you'd end up with two identical queues, not the same queue in two processes. So, multiprocessing extends things a bit, by adding a register_after_fork mechanism for objects to use when unpickling.** If you look at the source for Queue, you can see how it works.

但是您真的不需要知道如何钩住它.您可以像其他班级的酸洗一样钩住它.例如，这应该起作用:***

But you don't really need to know how it works to hook it; you can hook it the same way as any other class's pickling. For example, this should work:***

def __getstate__(self):
    return self.name, super(MyQueue, self).__getstate__()

def __setstate__(self, state):
    self.name, state = state
    super(MyQueue, self).__setstate__(state)

有关更多详细信息， pickle 文档说明了您使用的不同方法会影响您班级的腌制方式.

For more details, the pickle documentation explains the different ways you can influence how your class is pickled.

(如果它不起作用，并且我没有犯一个愚蠢的错误，那么您 do 必须至少了解一些如何钩住它的方法……但很可能只是找出是在_after_fork()之前还是之后做额外的工作，这只需要交换最后两行…)

(If it doesn't work, and I haven't made a stupid mistake… then you do have to know at least a little about how it works to hook it… but most likely just to figure out whether to do your extra work before or after the _after_fork(), which would just require swapping the last two lines…)

*我不确定在POSIX平台上是否真的可以保证使用简单的fork继承.在2.7和3.3上确实如此.但是有一个multiprocessing的分支在所有平台上都使用Windows样式的pickle来保持一致性，而另一个在OS X上使用混合使用以允许在单线程模式下使用CoreFoundation或类似的东西，显然，这样做是可行的.

* I'm not sure it's actually guaranteed to use simple fork inheritance on POSIX platforms. That happens to be true on 2.7 and 3.3. But there's a fork of multiprocessing that uses the Windows-style pickle-everything on all platforms for consistency, and another one that uses a hybrid on OS X to allow using CoreFoundation in single-threaded mode, or something like that, and it's clearly doable that way.

**实际上，我认为 Queue仅出于方便起见使用register_after_fork，并且可以在没有它的情况下对其进行重写……但这取决于Pipe在其在Windows上，或Lock和BoundedSemaphore在POSIX上.

** Actually, I think Queue is only using register_after_fork for convenience, and could be rewritten without it… but it's depending on the magic that Pipe does in its _after_fork on Windows, or Lock and BoundedSemaphore on POSIX.

***这是正确的，因为我从阅读源代码中偶然知道Queue是一种新型类，不会覆盖__reduce__或__reduce_ex，并且从不返回假值来自__getstate__.如果您不知道，则必须编写更多代码.

*** This is only correct because I happen to know, from reading the source, that Queue is a new-style class, doesn't override __reduce__ or __reduce_ex, and never returns a falsey value from __getstate__. If you didn't know that, you'd have to write more code.

这篇关于如何从多处理队列继承?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！