如何在python 3中将队列与将来的并发ThreadPoolExecutor一起使用? [英] How to use queue with concurrent future ThreadPoolExecutor in python 3?
问题描述
我正在使用简单的线程模块来执行并发作业.现在,我想利用并发期货模块的优势.有人可以给我一个使用并发库队列的例子吗?
I am using simple threading modules to do concurrent jobs. Now I would like to take advantages of concurrent futures modules. Can some put me a example of using a queue with concurrent library?
我收到TypeError:队列"对象不可迭代 我不知道如何迭代队列
I am getting TypeError: 'Queue' object is not iterable I dont know how to iterate queues
代码段:
def run(item):
self.__log.info(str(item))
return True
<queue filled here>
with concurrent.futures.ThreadPoolExecutor(max_workers = 100) as executor:
furtureIteams = { executor.submit(run, item): item for item in list(queue)}
for future in concurrent.futures.as_completed(furtureIteams):
f = furtureIteams[future]
print(f)
推荐答案
我建议这样:
def run(queue):
item = queue.get()
self.__log.info(str(item))
return True
<queue filled here>
workerThreadsToStart = 10
with concurrent.futures.ThreadPoolExecutor(max_workers = 100) as executor:
furtureIteams = { executor.submit(run, queue): index for intex in range(workerThreadsToStart)}
for future in concurrent.futures.as_completed(furtureIteams):
f = furtureIteams[future]
print(f)
您将要运行的问题是,队列被认为是无止境的,并且是将放入队列的线程与将项目移出队列的线程解耦的媒介.
The problem you will run in is that a queue is thought to be endless and as a medium to decouple the threads that put something into the queue and threads that get items out of the queue.
何时
- 您的物品数量有限
- 您可以一次计算所有项目
,然后并行处理它们,队列就没有意义了. 在这些情况下,ThreadPoolExecutor使队列过时.
and afterwards process them in parallel, a queue makes no sense. A ThreadPoolExecutor makes a queue obsolete in these cases.
我看了看ThreadPoolExecutor的来源:
I had a look at the ThreadPoolExecutor source:
def submit(self, fn, *args, **kwargs): # line 94
self._work_queue.put(w) # line 102
内部使用队列.
这篇关于如何在python 3中将队列与将来的并发ThreadPoolExecutor一起使用?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!