Python多处理队列失败 [英] Python multiprocessing Queue failure

问题描述

我创建了100个子进程

I create 100 child processes

proc_list = [
    Process(target = simulator, args=(result_queue,))
    for i in xrange(100)]

并启动它们

for proc in proc_list: proc.start()

每个进程经过一些处理后，将10000个元组放入result_queue(multiprocessing.Queue的实例).

Each process puts into the result_queue (instance of multiprocessing.Queue) 10000 tuples after doing some processing.

def simulate(alg_instance, image_ids, gamma, results,
                     simulations, sim_semaphore):
  (rs, qs, t_us) =  alg_instance.simulate_multiple(image_ids, gamma,
                                             simulations)
  all_tuples = zip(rs, qs, t_us)
  for result in all_tuples:
    results.put(result)
  sim_semaphore.release()

我应该(?)在队列中获得1000000个元组，但是经过各种运行，我得到了这些(样本)大小: 14912 19563 12952 13524 7487 18350 15986 11928 14281 14282 7317

I should be (?) getting 1000000 tuples at the queue, but after various runs I get these (sample) sizes: 14912 19563 12952 13524 7487 18350 15986 11928 14281 14282 7317

有什么建议吗?

推荐答案

我对多处理问题的解决方案几乎总是使用Manager对象.虽然公开的接口相同，但是基础实现要简单得多，并且bug更少.

My solution to multiprocessing issues is almost always to use the Manager objects. While the exposed interface is the same, the underlying implementation is much simpler and has less bugs.

from multiprocessing import Manager
manager = Manager()
result_queue = manager.Queue()

尝试一下，看看它是否不能解决您的问题.

Try it out and see if it doesn't fix your issues.

这篇关于Python多处理队列失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！