旧线程完成后如何启动新线程? [英] How to start a new thread when old one finishes?

问题描述

我在列表中有一个很大的数据集，需要做一些工作.

I have a large dataset in a list that I need to do some work on.

我想在给定的时间启动x个线程以在列表上工作，直到弹出该列表中的所有内容为止.

I want to start x amounts of threads to work on the list at any given time, until everything in that list has been popped.

我知道如何在给定的时间(通过使用thread1 .... thread20.start())启动x个线程(比如说20个)

I know how to start x amounts of threads (lets say 20) at a given time (by using thread1....thread20.start())

但是当前20个线程之一结束时如何使它启动一个新线程?因此在任何给定时间，有20个线程在运行，直到列表为空.

but how do I make it start a new thread when one of the first 20 threads finish? so at any given time there are 20 threads running, until the list is empty.

我到目前为止有什么:

class queryData(threading.Thread):
    def __init__(self,threadID):
        threading.Thread.__init__(self)
        self.threadID = threadID
    def run(self):
        global lst
        #Get trade from list
        trade = lst.pop()
        tradeId=trade[0][1][:6]
        print tradeId


thread1 = queryData(1)
thread1.start()

更新

我的代码如下:

for i in range(20):
    threads.append(queryData(i))
for thread in threads:
    thread.start()

while len(lst)>0:
    for iter,thread in enumerate(threads):
        thread.join()
        lock.acquire()
        threads[iter] = queryData(i)
        threads[iter].start()
        lock.release()

现在它从头开始启动20个线程...然后在一个线程结束时继续启动一个新线程.

Now it starts 20 threads in the beginning...and then keeps starting a new thread when one finishes.

但是，它效率不高，因为它要等待列表中的第一个完成，然后再等待第二个..依此类推.

However, it is not efficient, as it waits for the first one in the list to finish, and then the second..and so on.

有更好的方法吗?

基本上我需要:

-Start 20 threads:
-While list is not empty:
   -wait for 1 of the 20 threads to finish
   -reuse or start a new thread

推荐答案

正如我在评论中所建议的那样，我认为使用multiprocessing.pool.ThreadPool将是适当的-因为它可以处理您正在手动执行的大部分线程管理.您的代码自动.通过ThreadPool的apply_async()方法调用将所有线程排队等待处理后，唯一需要做的就是等待它们全部完成执行(除非您的代码可以做其他事情，当然).

As I suggested in a comment, I think using a multiprocessing.pool.ThreadPool would be appropriate — because it would handle much of the thread management you're manually doing in your code automatically. Once all the threads are queued-up for processing via ThreadPool's apply_async() method calls, the only thing that needs to be done is wait until they've all finished execution (unless there's something else your code could be doing, of course).

我已经将链接的答案中的代码翻译为另一个相关问题，因此它与您看起来的内容更加相似使其更易于在当前情况下理解.

I've translated the code in my linked answer to another related question so it's more similar to what you appear to be doing to make it easier to understand in the current context.

from multiprocessing.pool import ThreadPool
from random import randint
import threading
import time

MAX_THREADS = 5
print_lock = threading.Lock()  # Prevent overlapped printing from threads.

def query_data(trade):
    trade_id = trade[0][1][:6]
    time.sleep(randint(1, 3))  # Simulate variable working time for testing.
    with print_lock:
        print(trade_id)

def process_trades(trade_list):
    pool = ThreadPool(processes=MAX_THREADS)
    results = []
    while(trade_list):
        trade = trade_list.pop()
        results.append(pool.apply_async(query_data, (trade,)))

    pool.close()  # Done adding tasks.
    pool.join()  # Wait for all tasks to complete.

def test():
    trade_list = [[['abc', ('%06d' % id) + 'defghi']] for id in range(1, 101)]
    process_trades(trade_list)

if __name__ == "__main__":
    test()

这篇关于旧线程完成后如何启动新线程?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！