如何在 python 3 中将队列与并发未来的 ThreadPoolExecutor 一起使用? [英] How to use queue with concurrent future ThreadPoolExecutor in python 3?

问题描述

我正在使用简单的线程模块来执行并发作业.现在我想利用并发期货模块.有人能给我举一个使用队列和并发库的例子吗?

I am using simple threading modules to do concurrent jobs. Now I would like to take advantages of concurrent futures modules. Can some put me a example of using a queue with concurrent library?

我收到 TypeError: 'Queue' object is not iterable我不知道如何迭代队列

I am getting TypeError: 'Queue' object is not iterable I dont know how to iterate queues

代码片段:

 def run(item):
      self.__log.info(str(item))
      return True
<queue filled here>

with concurrent.futures.ThreadPoolExecutor(max_workers = 100) as executor:
        furtureIteams = { executor.submit(run, item): item for item in list(queue)}
        for future in concurrent.futures.as_completed(furtureIteams):
            f = furtureIteams[future]
            print(f)

推荐答案

我会建议这样的事情:

def run(queue):
      item = queue.get()
      self.__log.info(str(item))
      return True
<queue filled here>
workerThreadsToStart = 10
with concurrent.futures.ThreadPoolExecutor(max_workers = 100) as executor:
        furtureIteams = { executor.submit(run, queue): index for intex in range(workerThreadsToStart)}
        for future in concurrent.futures.as_completed(furtureIteams):
            f = furtureIteams[future]
            print(f)

您将遇到的问题是，队列被认为是无止境的，并且作为一种媒介来解耦将某些内容放入队列的线程和将项目从队列中取出的线程.

The problem you will run in is that a queue is thought to be endless and as a medium to decouple the threads that put something into the queue and threads that get items out of the queue.

当

1. 您的商品数量有限或
2. 您一次计算所有项目

然后并行处理它们，队列没有意义.在这些情况下，ThreadPoolExecutor 会使队列过时.

and afterwards process them in parallel, a queue makes no sense. A ThreadPoolExecutor makes a queue obsolete in these cases.

我查看了 ThreadPoolExecutor 源代码:

I had a look at the ThreadPoolExecutor source:

def submit(self, fn, *args, **kwargs): # line 94
    self._work_queue.put(w) # line 102

里面使用了一个队列.

这篇关于如何在 python 3 中将队列与并发未来的 ThreadPoolExecutor 一起使用?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！