Bash 脚本并行处理有限数量的命令 [英] Bash script processing limited number of commands in parallel

问题描述

我有一个看起来像这样的 bash 脚本:

#!/bin/bashwget LINK1 >/dev/null 2>&1wget LINK2 >/dev/null 2>&1wget LINK3 >/dev/null 2>&1wget LINK4 >/dev/null 2>&1#..#..wget LINK4000 >/dev/null 2>&1

但是处理每一行直到命令完成然后移动到下一行是非常耗时的，我想一次处理 20 行，然后当它们完成时再处理 20 行.

我想到了 wget LINK1 >/dev/null 2>&1 & 将命令发送到后台并继续，但是这里有 4000 行这意味着我将有性能问题，更不用说我应该同时启动多少进程的限制，所以这不是一个好主意.

我现在想到的一个解决方案是检查其中一个命令是否仍在运行，例如在 20 行之后我可以添加这个循环:

while [ $(ps -ef | grep KEYWORD | grep -v grep | wc -l) -gt 0 ];做睡觉 1完毕

当然在这种情况下我需要附加 &到最后！但我觉得这不是正确的做法.

那么我实际上如何将每 20 行组合在一起并等待它们完成，然后再转到下 20 行，该脚本是动态生成的，因此我可以在生成时对其进行任何我想做的数学运算，但确实如此不必使用 wget，这只是一个示例，因此任何特定于 wget 的解决方案都不会对我有任何好处.

解决方案

使用内置的wait:

process1 &过程 2 &过程3 &过程4 &等待过程5 &过程6 &过程7 &过程8 &等待

对于上面的例子，4 个进程 process1 ... process4 将在后台启动，shell 将等待这些进程完成后才开始下一组.

来自 GNU 手册:

<块引用>

wait [jobspec 或 pid ...]

等待每个进程ID pid或job规范jobspec指定的子进程退出，返回最后一个的退出状态命令等待.如果给出了作业规范，则作业中的所有进程正在等待.如果没有给出参数，所有当前活动的孩子进程被等待，返回状态为零.如果两者都没有jobspec 或 pid 指定 shell 的活动子进程，返回状态为 127.

I have a bash script that looks like this:

#!/bin/bash
wget LINK1 >/dev/null 2>&1
wget LINK2 >/dev/null 2>&1
wget LINK3 >/dev/null 2>&1
wget LINK4 >/dev/null 2>&1
# ..
# ..
wget LINK4000 >/dev/null 2>&1

But processing each line until the command is finished then moving to the next one is very time consuming, I want to process for instance 20 lines at once then when they're finished another 20 lines are processed.

I thought of wget LINK1 >/dev/null 2>&1 & to send the command to the background and carry on, but there are 4000 lines here this means I will have performance issues, not to mention being limited in how many processes I should start at the same time so this is not a good idea.

One solution that I'm thinking of right now is checking whether one of the commands is still running or not, for instance after 20 lines I can add this loop:

while [  $(ps -ef | grep KEYWORD | grep -v grep | wc -l) -gt 0 ]; do
sleep 1
done

Of course in this case I will need to append & to the end of the line! But I'm feeling this is not the right way to do it.

So how do I actually group each 20 lines together and wait for them to finish before going to the next 20 lines, this script is dynamically generated so I can do whatever math I want on it while it's being generated, but it DOES NOT have to use wget, it was just an example so any solution that is wget specific is not gonna do me any good.

解决方案

Use the wait built-in:

process1 &
process2 &
process3 &
process4 &
wait
process5 &
process6 &
process7 &
process8 &
wait

For the above example, 4 processes process1 ... process4 would be started in the background, and the shell would wait until those are completed before starting the next set.

From the GNU manual:

wait [jobspec or pid ...]

Wait until the child process specified by each process ID pid or job specification jobspec exits and return the exit status of the last command waited for. If a job spec is given, all processes in the job are waited for. If no arguments are given, all currently active child processes are waited for, and the return status is zero. If neither jobspec nor pid specifies an active child process of the shell, the return status is 127.

这篇关于Bash 脚本并行处理有限数量的命令的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！