并行运行子进程 [英] run subprocesses in parallel

问题描述

我有一个 python 脚本，它必须调用某个应用程序 3 次.这些调用应该是并行的，因为它们需要数小时才能完成并且不相互依赖.但是他们的脚本应该停止，直到所有这些都完成，然后做一些清理工作.

I have a python script which has to call a certain app 3 times. These calls should be parralel since they take hours to complete and arent dependant on eachother. But they script should halt until all of them are complete and then do some clean up work.

这是一些代码:

#do some stuff

for work in worklist:   # these should run in parralel
    output=open('test.txt','w')
    subprocess.call(work,stdout=output,stderr=output)
    output.close()

# wait for subprocesses to finish

# cleanup

所以我基本上想在 parrelel 中运行这个命令，同时将它的输出捕获到一个文件中.完成所有实例后，我想继续脚本

so I basically want to run this command in parrelel while capturing its output to a file. once all instances are done I want to continue the script

推荐答案

subprocess.call() 正在阻塞.这意味着，每次调用都必须等待子进程完成才能继续.

subprocess.call() is blocking. That means, each call must wait for the child process to finish before continuing.

您想要的是将参数传递给 subprocess.Popen 构造函数.这样，您的子进程将在不阻塞的情况下启动.

What you want is to pass your arguments to subprocess.Popen constructor, instead. That way, your child process would be started without blocking.

稍后，您可以通过调用 Popen.communicate() 或 Popen.wait() 将这些子进程连接在一起.

Later on, you can join these child processes together by calling Popen.communicate() or Popen.wait().

child_processes = []
for work, filename in worklist:
    with io.open(filename, mode='wb') as out:
        p = subprocess.Popen(work, stdout=out, stderr=out)
        child_processes.append(p)    # start this one, and immediately return to start another

# now you can join them together
for cp in child_processes:
    cp.wait()                         # this will block on each child process until it exits

附言您是否查看过关于 subprocess 模块的 Python 文档?

P.S. Have you looked into Python's documentation on the subprocess module?

这篇关于并行运行子进程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！