使用python在队列中逐个进程运行进程 [英] Run process after process in a queue using python

问题描述

我有500个要通过python脚本运行的进程的队列，我想并行运行每N个进程.

I have a queue of 500 processes that I want to run through a python script, I want to run every N processes in parallel.

到目前为止，我的python脚本做了什么: 它并行运行N个进程，等待所有进程终止，然后运行下一个N个文件.

What my python script does so far: It runs N processes in parallel, waits for all of them to terminate, then runs the next N files.

我需要做的是: N个进程之一完成后，队列中的另一个进程将自动启动，而无需等待其余进程终止.

What I need to do: When one of the N processes is finished, another process from the queue is automatically started, without waiting for the rest of the processes to terminate.

注意:我不知道每个进程将花费多少时间，因此我无法安排某个进程在特定时间运行.

Note: I do not know how much time each process will take, so I can't schedule a process to run at a particular time.

以下是我的代码. 我目前正在使用subprocess.Popen，但不仅限于此.

Following is the code that I have. I am currently using subprocess.Popen, but I'm not limited to its use.

for i in range(0, len(queue), N):
    batch = []
    for _ in range(int(jobs)):
        batch.append(queue.pop(0))
    for process in batch:
        p = subprocess.Popen([process])
        ps.append(p)
    for p in ps:
        p.communicate()

推荐答案

我认为这应该可行:

import subprocess
import time


def check_for_done(l):
    for i, p in enumerate(l):
        if p.poll() is not None:
            return True, i
    return False, False


processes = list()
N = 5
queue = list()
for process in queue:
    p = subprocess.Popen(process)
    processes.append(p)
    if len(processes) == N:
        wait = True
        while wait:
            done, num = check_for_done(processes)

            if done:
                processes.pop(num)
                wait = False
            else:
                time.sleep(0.5) # set this so the CPU does not go crazy

因此，您有一个活动的进程列表，并且check_for_done函数遍历该进程列表，如果该子进程尚未完成，则返回None；如果是，则返回一个返回码.因此，当返回某些内容时，应该执行此操作(不知道该操作是否成功).然后，从列表中删除该进程，以允许循环添加另一个进程.

So you have an active process list, and the check_for_done function loops through it, the subprocess returns None if it is not finished and it returns a return code if it is. So when something is returned it should be done (without knowing if it was successful or not). Then you remove that process from the list allowing for the loop to add another one.

这篇关于使用python在队列中逐个进程运行进程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！