在多个进程之间共享一个结果队列 [英] Sharing a result queue among several processes

问题描述

multiprocessing 模块的文档展示了如何将队列传递给以 multiprocessing.Process 启动的进程.但是如何与以 apply_async 启动的异步工作进程共享队列?我不需要动态加入或其他任何东西，只是让工人(反复)向基地报告他们的结果的一种方式.

The documentation for the multiprocessing module shows how to pass a queue to a process started with multiprocessing.Process. But how can I share a queue with asynchronous worker processes started with apply_async? I don't need dynamic joining or anything else, just a way for the workers to (repeatedly) report their results back to base.

import multiprocessing
def worker(name, que):
    que.put("%d is done" % name)

if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=3)
    q = multiprocessing.Queue()
    workers = pool.apply_async(worker, (33, q))

这失败了:RuntimeError:队列对象只能通过继承在进程之间共享.我理解这意味着什么，并且我理解继承而不是要求酸洗/取消酸洗(以及所有特殊的 Windows 限制)的建议.但是我如何做以一种有效的方式传递队列?我找不到示例，我尝试了几种以各种方式失败的替代方法.请帮忙?

This fails with: RuntimeError: Queue objects should only be shared between processes through inheritance. I understand what this means, and I understand the advice to inherit rather than require pickling/unpickling (and all the special Windows restrictions). But how do I pass the queue in a way that works? I can't find an example, and I've tried several alternatives that failed in various ways. Help please?

推荐答案

尝试使用 multiprocessing.Manager 来管理您的队列并使其可供不同的工作人员访问.

Try using multiprocessing.Manager to manage your queue and to also make it accessible to different workers.

import multiprocessing
def worker(name, que):
    que.put("%d is done" % name)

if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=3)
    m = multiprocessing.Manager()
    q = m.Queue()
    workers = pool.apply_async(worker, (33, q))

这篇关于在多个进程之间共享一个结果队列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！