多处理队列最大大小限制为 32767 [英] Multiprocessing Queue maxsize limit is 32767

问题描述

我正在尝试使用多处理编写 Python 2.6 (OSX) 程序，并且我想用超过默认的 32767 个项目来填充队列.

I'm trying to write a Python 2.6 (OSX) program using multiprocessing, and I want to populate a Queue with more than the default of 32767 items.

from multiprocessing import Queue
Queue(2**15) # raises OSError

Queue(32767) 工作正常，但任何更大的数字(例如 Queue(32768))都会失败，并出现 OSError: [Errno 22] Invalid argument

Queue(32767) works fine, but any higher number (e.g. Queue(32768)) fails with OSError: [Errno 22] Invalid argument

这个问题有解决办法吗?

Is there a workaround for this issue?

推荐答案

一种方法是使用自定义类包装您的 multiprocessing.Queue(仅在生产者端，或从消费者透明看法).使用它，您可以将要分派到您正在包装的 Queue 对象的项目排队，并且仅将本地队列(Python list() 对象)中的内容提供给multiprocess.Queue 随着空间变得可用，当 Queue 已满时进行异常处理以节流.

One approach would be to wrap your multiprocessing.Queue with a custom class (just on the producer side, or transparently from the consumer perspective). Using that you would queue up items to be dispatched to the Queue object that you're wrapping, and only feed things from the local queue (Python list() object) into the multiprocess.Queue as space becomes available, with exception handling to throttle when the Queue is full.

这可能是最简单的方法，因为它对您的其余代码的影响应该最小.自定义类的行为应该像队列一样，同时将底层的 multiprocessing.Queue 隐藏在您的抽象后面.

That's probably the easiest approach since it should have the minimum impact on the rest of your code. The custom class should behave just like a Queue while hiding the underlying multiprocessing.Queue behind your abstraction.

(一种方法可能是让您的生产者使用线程，一个线程来管理从线程 Queue 到您的 multiprocessing.Queue 和任何其他线程实际上只是喂入线程Queue).

(One approach might be to have your producer use threads, one thread to manage the dispatch from a threading Queue to your multiprocessing.Queue and any other threads actually just feeding the threading Queue).

这篇关于多处理队列最大大小限制为 32767的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！