并行bash脚本处理命令 [英] Bash script processing commands in parallel

问题描述

我有一个bash脚本，如下所示：

 ＃！/斌/庆典
wget的LINK1＆GT;的/ dev / null的2  - ;＆放大器; 1
wget的LINK2＆GT;的/ dev / null的2  - ;＆放大器; 1
wget的LINK3＆GT;的/ dev / null的2  - ;＆放大器; 1
wget的LINK4＆GT;的/ dev / null的2  - ;＆放大器; 1
＃..
＃..
wget的LINK4000＆GT;的/ dev / null的2  - ;＆放大器; 1
 

但处理每一行，直至完成然后移动到下一个非常耗时的命令，我要当他们完成另外20行处理一次，然后20日线的实例来处理。

我觉得 wget的LINK1制造＆gt;的/ dev / null的2 - ;＆放大器; 1安培; 将命令发送到后台并进行，但也有4000行在这里，这意味着我将有性能问题，更何况，我应该多少进程在同一时间开始被限制，因此这不是一个好主意。

一个解决方案，我在想，现在正在检查一个命令是否仍在运行与否，例如后20行，我可以添加这个循环：

 ，而[$（PS -ef | grep的导航| grep的-v grep的|厕所-l）-gt 0];做
睡眠1
DONE
 

当然，在这种情况下，我需要追加＆安培;该行的结尾！但是我觉得这是不是这样做的正确方法。

那么，如何做我其实组各20条线一起，等待他们才去到下一个20行完成，是动态生成的这个剧本，所以我可以做我想做的是什么数学的生成，而它，但它确实不必使用wget，这只是一个例子，因此任何解决方案，具体的wget是不是要去我好。

解决方案

使用等内置：

 ＆过程1安培;
过程2＆安培;
process3＆安培;
process4＆安培;
等待
process5＆安培;
process6＆安培;
process7＆安培;
process8＆安培;
等待
 

对于上面的例子中，4个进程过程1 .. process4 将在后台启动，壳会等到这些都开始下一组之前完成的。

从手动：

 等待[JOBSPEC或PID ...]
 

  
  

等到每个进程指定的ID PID或工作规范子进程退出JOBSPEC并返回最后的退出状态
  命令等待。如果作业规范，给出所有进程的工作
  正在等待。如果没有给出参数，所有当前活动的孩子
  过程等待，并返回状态是零。如果没有
  JOBSPEC也不指定PID的外壳活动的子过程中，
  返回状态是127。

I have a bash script that looks like this:

#!/bin/bash
wget LINK1 >/dev/null 2>&1
wget LINK2 >/dev/null 2>&1
wget LINK3 >/dev/null 2>&1
wget LINK4 >/dev/null 2>&1
# ..
# ..
wget LINK4000 >/dev/null 2>&1

But processing each line until the command is finished then moving to the next one is very time consuming, I want to process for instance 20 lines at once then when they're finished another 20 lines are processed.

I thought of wget LINK1 >/dev/null 2>&1 & to send the command to the background and carry on, but there are 4000 lines here this means I will have performance issues, not to mention being limited in how many processes I should start at the same time so this is not a good idea.

One solution that I'm thinking of right now is checking whether one of the commands is still running or not, for instance after 20 lines I can add this loop:

while [  $(ps -ef | grep KEYWORD | grep -v grep | wc -l) -gt 0 ]; do
sleep 1
done

Of course in this case I will need to append & to the end of the line! But I'm feeling this is not the right way to do it.

So how do I actually group each 20 lines together and wait for them to finish before going to the next 20 lines, this script is dynamically generated so I can do whatever math I want on it while it's being generated, but it DOES NOT have to use wget, it was just an example so any solution that is wget specific is not gonna do me any good.

解决方案

Use the wait built-in:

process1 &
process2 &
process3 &
process4 &
wait
process5 &
process6 &
process7 &
process8 &
wait

For the above example, 4 processes process1 .. process4 would be started in the background, and the shell would wait until those are completed before starting the next set ..

From the manual:

wait [jobspec or pid ...]

Wait until the child process specified by each process ID pid or job specification jobspec exits and return the exit status of the last command waited for. If a job spec is given, all processes in the job are waited for. If no arguments are given, all currently active child processes are waited for, and the return status is zero. If neither jobspec nor pid specifies an active child process of the shell, the return status is 127.

这篇关于并行bash脚本处理命令的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！