Bash:在多个内核上运行相同的程序 [英] Bash: Running the same program over multiple cores

问题描述

我可以访问一台可以访问10个内核的机器-我想实际使用它们.我习惯在自己的机器上做的事情是这样的:

I have access to a machine where I have access to 10 of the cores -- and I would like to actually use them. What I am used to doing on my own machine would be something like this:

for f in *.fa; do
  myProgram (options) "./$f" "./$f.tmp"
done

我要执行10个文件，我们称它们为blah00.fa，blah01.fa，... blah09.fa.

I have 10 files I'd like to do this on -- let's call them blah00.fa, blah01.fa, ... blah09.fa.

这种方法的问题是myProgram一次仅使用1个内核，并且在多核计算机上这样做，我将一次使用1个内核，因此我不会使用10次我的机器将其发挥到最大能力.

The problem with this approach is that myProgram only uses 1 core at a time, and doing it like this on the multi-core machine I'd be using 1 core at a time 10 times, so I wouldn't be using my mahcine to its max capability.

如何更改脚本，以使其同时运行所有10个.fa文件?我查看了在bash中跨多个内核运行循环的过程但我无法从中获得命令来完成我想要的事情.

How could I change my script so that it runs all 10 of my .fa files at the same time? I looked at Run a looped process in bash across multiple cores but I couldn't get the command from that to do what I wanted exactly.

推荐答案

您可以使用

for f in *.fa; do
    myProgram (options) "./$f" "./$f.tmp" &
done
wait

这将并行启动所有工作，然后等到它们全部完成后再继续.如果您的工作多于核心，那么您将启动所有这些工作，并让您的OS调度程序担心换出进程.

which would start all of you jobs in parallel, then wait until they all complete before moving on. In the case where you have more jobs than cores, you would start all of them and let your OS scheduler worry about swapping processes in an out.

一种修改是一次启动10个作业

One modification is to start 10 jobs at a time

count=0
for f in *.fa; do
    myProgram (options) "./$f" "./$f.tmp" &
    (( count ++ ))        
    if (( count = 10 )); then
        wait
        count=0
    fi
done

，但这不如使用parallel，因为您无法在旧作业完成时启动新作业，并且在设法启动10个作业之前也无法检测到较旧的作业是否已完成. wait允许您等待单个特定进程或所有后台进程，但不会让您知道任意一组后台进程的完成时间.

but this is inferior to using parallel because you can't start new jobs as old ones finish, and you also can't detect if an older job finished before you manage to start 10 jobs. wait allows you to wait on a single particular process or all background processes, but doesn't let you know when any one of an arbitrary set of background processes complete.

这篇关于Bash:在多个内核上运行相同的程序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！