如何将线程池中的线程数限制为无限可迭代? [英] How to limit the nr of threads in a thread pool for infinite iterable?

问题描述

假设我们要无限迭代所有自然数，并为每个数字打开一个线程，直到一个极限.由于存在无限的自然数，因此该列表的作用类似于生成器，那么在生成数字时如何限制开放线程的数量呢?

Suppose we want to iterate infinitely through all the natural numbers and open a thread for each number up to a limit. Since there are infinite natural number this list acts like a generator, so how can we keep a limit on the number of open threads while generating the numbers?

我正在尝试类似的事情:

I'm trying something like:

from multiprocessing import Pool

def func(i):
    """ Do something with i"""

p = Pool(10)
i = 0
while True:
    p.imap_unordered(func,i)
    i += 1

我尝试了许多其他方法(线程池，信号量等)，但它们似乎都忽略了线程的最大nr值.我想做一些事情，打开MAX_THREADS个线程，每当一个线程完成时，它就会再次迭代并开始一个新的线程.我该如何实现?

I tried many other different ways (threadpools, semaphores, etc) but all of them seem to ignore the max nr of threads. I want to make something take opens up MAX_THREADS threads and everytime a thread finishes, it iterates again and start a new one. How can i achieve this?

推荐答案

您要使用 ThreadPoolExecutor .构造函数的第一个参数是max_workers，它确定它可以使用多少个线程.

You want to use a ThreadPoolExecutor. The first parameter to the constructor is max_workers, which determines how many threads it can use.

因此，您最终得到的是这样的东西:

So you end up with something like this:

with ThreadPoolExecutor(max_workers=n) as pool:
    future = pool.submit(function, *args)
future.result() # outputs result

返回的值是 Future ，这样您就可以在将来完成操作后访问结果. with块结束时，ThreadPoolExecutor等待其所有工作程序完成，因此您可以在该块结束后安全地访问结果.

The returned value is a Future, so you can access the result once the future is completed. When the with block ends, the ThreadPoolExecutor waits for all of its workers to finish, so you can safely access the results after the end of the block.

这是ThreadPoolExecutor的简单示例:

from concurrent.futures import ThreadPoolExecutor
from time import sleep

def fn(delay, result):
    sleep(x)
    print(result)

with ThreadPoolExecutor(max_workers=2) as pool:
    pool.submit(fn, 3, "last")
    pool.submit(fn, 1, "first")
    pool.submit(fn, 1, "middle")
# This should output "first", then "middle", then "last"

这篇关于如何将线程池中的线程数限制为无限可迭代?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！