Python多重处理:RuntimeError:“队列对象仅应通过继承在进程之间共享". [英] Python multiprocessing: RuntimeError: "Queue objects should only be shared between processes through inheritance"

问题描述

我知道multiprocessing.Manager()以及如何将其用于创建共享对象.特别是可以在工作人员之间共享的队列.有这个问题，此问题和此问题.

I am aware of multiprocessing.Manager() and how it can be used to create shared objects. In particular, queues which can be shared among workers. There is this question, this question, and this question.

但是，这些链接没有提到为什么我们可以使用继承在进程之间共享.据我了解，在这种情况下，仍然只能复制队列.

However, these links don't mention why we can use inheritance for sharing between processes. As I understand, a queue can still only be copied in this case.

推荐答案

python中的Queue实现依赖于系统pipe来将数据从一个进程传输到另一个进程，而某些semaphores则将数据传输到另一个进程以保护读取和读取.在此pipe上写.

The Queue implementation in python relies on a system pipe to transmit the data from one process to another and some semaphores to protect the read and write on this pipe.

由于操作系统的限制，pipe在进程中被视为打开文件，并且只能在生成时与子进程共享.
对于早期版本的python，semaphores也被视为仅应在生成时共享的文件(至少在基于UNIX的系统中).

The pipe is handled as an open file in the process and can only be shared with a subprocess at spawning time, because of OS constraints.
The semaphores are also treated as files that should only be shared at spawning time, at least in UNIX based system, for early version of python.

由于这两个子对象通常无法共享，因此Queue一旦启动就不能被腌制并发送到子进程.

As these 2 sub objects cannot be shared in general, the Queue cannot be pickled and sent to a subprocess once it has been started.

但是，对于某些操作系统和最新版本的python，可以共享Connection并创建可共享的Semaphore.因此，从理论上讲，您可以创建自己的Queue，可以在进程之间共享.但是它涉及很多黑客行为，而且可能不是很安全.

However, for some OS and recent version of python, it is possible to share the Connection and to create sharable Semaphore. Thus, you could in theory create your own Queue that can be shared between processes. But it involves a lot of hacks and might not be very secured.

这篇关于Python多重处理:RuntimeError:“队列对象仅应通过继承在进程之间共享".的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！