线程队列工作示例 [英] Threading queue working example

问题描述

如何在以下代码中将打开线程的最大数量限制为20个?我知道过去也曾提出过类似的问题，但我特别想知道如何最好地使用Queue和可行的示例来完成此工作.

How can I limit the maximum of open threads to 20 in the following code? I'm aware that there have been some similar questions asked in the past, but I specifically want to know how this is best done with a Queue and with a working example if possible.

    # b is a list with 10000 items
    threads = [threading.Thread(target=targetFunction, args=(ptf,anotherarg)) for ptf in b]
    for thread in threads:
        thread.start()

    for thread in threads:
        thread.join()

推荐答案

执行此操作的简单方法是使用queue.Queue进行工作，并使用for _ in range(MAXTHREADS): threading.Thread(target=f, args=(the_queue,)).start()启动线程.但是，通过子类化Thread，我发现这更容易阅读.您的里程可能会有所不同.

The simple way to do this is with a queue.Queue for the work and starting the threads with for _ in range(MAXTHREADS): threading.Thread(target=f, args=(the_queue,)).start(). I find this easier to read by subclassing Thread, however. Your mileage may vary.

import threading
import queue

class Worker(threading.Thread):
    def __init__(self, q, other_arg, *args, **kwargs):
        self.q = q
        self.other_arg = other_arg
        super().__init__(*args, **kwargs)
    def run(self):
        while True:
            try:
                work = self.q.get(timeout=3)  # 3s timeout
            except queue.Empty:
                return
            # do whatever work you have to do on work
            self.q.task_done()

q = queue.Queue()
for ptf in b:
    q.put_nowait(ptf)
for _ in range(20):
    Worker(q, otherarg).start()
q.join()  # blocks until the queue is empty.

如果您坚持使用函数，建议您将targetFunction包裹在知道如何从队列中获取的内容上.

If you're insistent about using a function, I'd suggest wrapping your targetFunction with something that knows how to get from the queue.

def wrapper_targetFunc(f, q, somearg):
    while True:
        try:
            work = q.get(timeout=3)  # or whatever
        except queue.Empty:
            return
        f(work, somearg)
        q.task_done()

q = queue.Queue()
for ptf in b:
    q.put_nowait(ptf)
for _ in range(20):
    threading.Thread(target=wrapper_targetFunc,
                     args=(targetFunction, q, otherarg)).start()
q.join()

这篇关于线程队列工作示例的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！