如何在 Python 中使用多处理队列? [英] How to use multiprocessing queue in Python?

问题描述

我在试图理解多处理队列如何在 python 上工作以及如何实现它时遇到了很多麻烦.假设我有两个 Python 模块可以访问共享文件中的数据，我们将这两个模块称为编写器和读取器.我的计划是让读取器和写入器将请求放入两个单独的多处理队列中，然后让第三个进程在循环中弹出这些请求并执行.

I'm having much trouble trying to understand just how the multiprocessing queue works on python and how to implement it. Lets say I have two python modules that access data from a shared file, let's call these two modules a writer and a reader. My plan is to have both the reader and writer put requests into two separate multiprocessing queues, and then have a third process pop these requests in a loop and execute as such.

我的主要问题是我真的不知道如何正确实现 multiprocessing.queue，你不能真正为每个进程实例化对象，因为它们将是单独的队列，你如何确保所有进程都与共享进程相关队列(或在这种情况下，队列)

My main problem is that I really don't know how to implement multiprocessing.queue correctly, you cannot really instantiate the object for each process since they will be separate queues, how do you make sure that all processes relate to a shared queue (or in this case, queues)

推荐答案

我的主要问题是我真的不知道如何正确实现 multiprocessing.queue，你不能真正为每个进程实例化对象，因为它们将是单独的队列，你如何确保所有进程都与共享进程相关队列(或在这种情况下，队列)

My main problem is that I really don't know how to implement multiprocessing.queue correctly, you cannot really instantiate the object for each process since they will be separate queues, how do you make sure that all processes relate to a shared queue (or in this case, queues)

这是一个简单的读写器共享一个队列的例子……写入器向读取器发送一堆整数；当写入器用完数字时，它会发送DONE"，让读取器知道跳出读取循环.

This is a simple example of a reader and writer sharing a single queue... The writer sends a bunch of integers to the reader; when the writer runs out of numbers, it sends 'DONE', which lets the reader know to break out of the read loop.

from multiprocessing import Process, Queue
import time
import sys

def reader_proc(queue):
    ## Read from the queue; this will be spawned as a separate Process
    while True:
        msg = queue.get()         # Read from the queue and do nothing
        if (msg == 'DONE'):
            break

def writer(count, queue):
    ## Write to the queue
    for ii in range(0, count):
        queue.put(ii)             # Write 'count' numbers into the queue
    queue.put('DONE')

if __name__=='__main__':
    pqueue = Queue() # writer() writes to pqueue from _this_ process
    for count in [10**4, 10**5, 10**6]:             
        ### reader_proc() reads from pqueue as a separate process
        reader_p = Process(target=reader_proc, args=((pqueue),))
        reader_p.daemon = True
        reader_p.start()        # Launch reader_proc() as a separate python process

        _start = time.time()
        writer(count, pqueue)    # Send a lot of stuff to reader()
        reader_p.join()         # Wait for the reader to finish
        print("Sending {0} numbers to Queue() took {1} seconds".format(count, 
            (time.time() - _start)))

这篇关于如何在 Python 中使用多处理队列?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！