限制并行工作的线程数 [英] limit number of threads working in parallel

问题描述

am提供了将文件从本地计算机复制到远程创建线程以并行执行sftp的功能

am making a function to copy file from local machine to remote creating thread to do sftp in parallel

def copyToServer():
    //does  copy file given host name and credentials

for i in hostsList:
    hostname = i
    username = defaultLogin
    password = defaultPassword
    thread = threading.Thread(target=copyToServer, args=(hostname, username, password, destPath, localPath))
    threadsArray.append(thread)
    thread.start()

这将创建线程并开始并行复制，但由于服务器总数过多，我想限制它一次只能处理50个线程

this creates thread and does start copying in parallel but i want to limit it to process like 50 threads at a time as total number of servers could be too many

推荐答案

您需要调整代码以共享并跟踪共同的值.

You need to adjust your code to share and keep track of a common value.

这可以通过信号量对象来完成.该对象拥有一个内部计数器，每个线程都尝试获取它.如果计数器大于您定义的最大值，则该线程无法获取一个，并且将被阻塞，直到一个空闲.

This could be done with a Semaphore Object. The object holds an internal counter and every thread try to acquire it. If the counter is bigger than your defined maximum, the thread can't acquire one and will be blocked until one gets free.

一个简短的示例显示了最多5个并行线程，其中一半线程立即执行，其他线程被阻塞并等待:

A short example shows for a maximum of 5 threads in parallel, that one half of the threads are executed instantly and the others are blocked and wait:

import threading
import time

maxthreads = 5
sema = threading.Semaphore(value=maxthreads)
threads = list()

def task(i):
    sema.acquire()
    print "start %s" % (i,)
    time.sleep(2)
    sema.release()

for i in range(10):
    thread = threading.Thread(target=task,args=(str(i)))
    threads.append(thread)
    thread.start()

输出

start 0
start 1
start 2
start 3
start 4

几秒钟后，第一个线程完成，然后执行下一个线程

and after some seconds the first threads are finished the next threads are executed

start 5
start 6
start 7
start 8
start 9

这篇关于限制并行工作的线程数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！