Bash脚本并行处理有限数量的命令 [英] Bash script processing limited number of commands in parallel

问题描述

我有一个如下所示的bash脚本:

I have a bash script that looks like this:

#!/bin/bash
wget LINK1 >/dev/null 2>&1
wget LINK2 >/dev/null 2>&1
wget LINK3 >/dev/null 2>&1
wget LINK4 >/dev/null 2>&1
# ..
# ..
wget LINK4000 >/dev/null 2>&1

但是处理每一行直到命令完成然后转移到下一行非常耗时，我想一次处理例如20行，然后当它们完成时再处理20行.

But processing each line until the command is finished then moving to the next one is very time consuming, I want to process for instance 20 lines at once then when they're finished another 20 lines are processed.

我想到了wget LINK1 >/dev/null 2>&1 &将命令发送到后台并继续执行，但是这里有4000行，这意味着我将遇到性能问题，更不用说我应该同时启动多少个进程了所以这不是一个好主意.

I thought of wget LINK1 >/dev/null 2>&1 & to send the command to the background and carry on, but there are 4000 lines here this means I will have performance issues, not to mention being limited in how many processes I should start at the same time so this is not a good idea.

我现在正在考虑的一种解决方案是检查其中一个命令是否仍在运行，例如，在20行之后，我可以添加以下循环:

One solution that I'm thinking of right now is checking whether one of the commands is still running or not, for instance after 20 lines I can add this loop:

while [  $(ps -ef | grep KEYWORD | grep -v grep | wc -l) -gt 0 ]; do
sleep 1
done

在这种情况下，我当然需要附加&到行尾！但是我感觉这不是正确的方法.

Of course in this case I will need to append & to the end of the line! But I'm feeling this is not the right way to do it.

所以我实际上如何将每20行组合在一起并等待它们完成，然后再转到下20行，该脚本是动态生成的，因此我可以在生成脚本时对其进行所需的任何数学运算，但是确实可以不必使用wget，这只是一个示例，因此任何特定于wget的解决方案都不会给我带来任何好处.

So how do I actually group each 20 lines together and wait for them to finish before going to the next 20 lines, this script is dynamically generated so I can do whatever math I want on it while it's being generated, but it DOES NOT have to use wget, it was just an example so any solution that is wget specific is not gonna do me any good.

推荐答案

使用内置的wait:

process1 &
process2 &
process3 &
process4 &
wait
process5 &
process6 &
process7 &
process8 &
wait

对于上面的示例，将在后台启动4个进程process1 ... process4，shell将等待直到完成这些进程，然后再开始下一个进程.

For the above example, 4 processes process1 ... process4 would be started in the background, and the shell would wait until those are completed before starting the next set.

从 GNU手册:

wait [jobspec or pid ...]

等待直到由每个进程ID pid或作业规范jobspec指定的子进程退出并返回最后一个的退出状态 命令等待.如果给出了工作说明，则该工作中的所有流程 等待着.如果未提供任何参数，则所有当前活动的子级 等待进程，返回状态为零.如果两者都不 jobspec或pid指定shell的活动子进程， 返回状态为127.

Wait until the child process specified by each process ID pid or job specification jobspec exits and return the exit status of the last command waited for. If a job spec is given, all processes in the job are waited for. If no arguments are given, all currently active child processes are waited for, and the return status is zero. If neither jobspec nor pid specifies an active child process of the shell, the return status is 127.

这篇关于Bash脚本并行处理有限数量的命令的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！