Python 多处理:RuntimeError:“队列对象只能通过继承在进程之间共享" [英] Python multiprocessing: RuntimeError: "Queue objects should only be shared between processes through inheritance"

问题描述

我知道 multiprocessing.Manager() 以及如何使用它来创建共享对象.特别是可以在工作人员之间共享的队列.有这个 问题，这个问题，这个问题.

I am aware of multiprocessing.Manager() and how it can be used to create shared objects. In particular, queues which can be shared among workers. There is this question, this question, and this question.

但是，这些链接没有提到为什么我们可以使用继承在进程之间共享.据我了解，仍然只能在这种情况下复制队列.

However, these links don't mention why we can use inheritance for sharing between processes. As I understand, a queue can still only be copied in this case.

推荐答案

python 中的 Queue 实现依赖于系统 pipe 将数据从一个进程传输到另一个进程和一些 semaphores 来保护这个 pipe 上的读写.

The Queue implementation in python relies on a system pipe to transmit the data from one process to another and some semaphores to protect the read and write on this pipe.

管道在进程中作为打开文件处理，并且由于操作系统限制，只能在生成时与子进程共享.
semaphores 也被视为只应在生成时共享的文件，至少在基于 UNIX 的系统中，对于早期版本的 python.

The pipe is handled as an open file in the process and can only be shared with a subprocess at spawning time, because of OS constraints.
The semaphores are also treated as files that should only be shared at spawning time, at least in UNIX based system, for early version of python.

由于这两个子对象一般不能共享，Queue一旦启动就不能被pickle并发送到子进程.

As these 2 sub objects cannot be shared in general, the Queue cannot be pickled and sent to a subprocess once it has been started.

但是，对于某些操作系统和最新版本的 python，可以共享 Connection 并创建可共享的 Semaphore.因此，理论上您可以创建自己的 Queue 可以在进程之间共享.但它涉及很多黑客攻击，可能不是很安全.

However, for some OS and recent version of python, it is possible to share the Connection and to create sharable Semaphore. Thus, you could in theory create your own Queue that can be shared between processes. But it involves a lot of hacks and might not be very secured.

这篇关于Python 多处理:RuntimeError:“队列对象只能通过继承在进程之间共享"的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！