多线程bash编程 - 广义的方法？ [英] Multi-threaded BASH programming - generalized method?

问题描述

好吧，我是在所有的演示运行 POV-Ray的，但POV的还是单线程和止跌'吨利用多个芯。于是，我开始考虑在BASH的解决方案。

我写这需要一个命令列表并运行它们的子炮弹的指定号码的一般功能。其实，这工作，但我不喜欢它处理的<击>访问下一个命令线程安全 多进程的方式的方式：

• 这需要作为一个参数，使用命令（每行）的文件，


• 要获得下一个命令，每个进程（线程）将：
  
  
  
  • 等待，直到它可以创建一个锁定文件，用： LN $ CMDFILE $锁文件
  
  
  • 从文件中读取命令，
  
  
  • ：修改$ CMDFILE通过移除第一线，
  
  
  • 删除$锁文件。
  
  

有一个更清洁的方式做到这一点？我不能让子壳正确读取从FIFO一行。

---

顺便说一句，这点是要提高什么我可以做一个bash命令行，并没有发现非bash的解决方案。我倾向于在命令行中执行大量的复杂的任务，并希望在工具箱中的另一个工具。

同时，这里的处理正从该文件的下一行的功能。正如你所看到的，它修改每次读取/删除订单时间磁盘上的文件。这就是看起来的hackish，但我不会想出什么都好，因为FIFO的没有工作的W / O的 setvbuf用来（）的在bash。

 ＃
＃获取/删除文件的第一行，用LOCK作为信号（带
＃碰撞睡眠不足）。返回在标准输出的文本，
＃返回零成功，非零值时，文件是空的。
＃
parallel__nextLine（）
{
  本地线路其余文件= $ 1锁定= $ 16  ＃等待锁...
  直到LN$ {}文件$ {}锁2  - ;的/ dev / null的
  不要睡1
     [-s$ {}文件] ||返回$？
  DONE  ＃打开，读取一个行救休息回文件：
  EXEC 3版;$文件
  读线474;及3;休息= $（猫＆LT;及3）
  EXEC 3';＆放大器;  -   ＃最后一行后，确保文件是空的：
  （[-z$休息] ||回声$休息）＆GT; $ {}文件  ＃删除锁和回归读行：
  RM -f$ {}锁定
  [-n$线]＆放大器;＆安培;回声$线
}
 

解决方案

  #adjust这些要求
args_per_proc = 1＃1罚款长时间运行的任务
procs_in_parallel = 4xargs的-n $ args_per_proc -P $ procs_in_parallel＆的povray LT;名单
 

请注意即将推出的coreutils会自动确定 NPROC 命令
然后可被传递给-P可用的处理单元的数目

Ok, I was running POV-Ray on all the demos, but POV's still single-threaded and wouldn't utilize more than one core. So, I started thinking about a solution in BASH.

I wrote a general function that takes a list of commands and runs them in the designated number of sub-shells. This actually works but I don't like the way it handles accessing the next command in a thread-safe multi-process way:

• It takes, as an argument, a file with commands (1 per line),
• To get the "next" command, each process ("thread") will:
  • Waits until it can create a lock file, with: ln $CMDFILE $LOCKFILE
  • Read the command from the file,
  • Modifies $CMDFILE by removing the first line,
  • Removes the $LOCKFILE.

Is there a cleaner way to do this? I couldn't get the sub-shells to read a single line from a FIFO correctly.

---

Incidentally, the point of this is to enhance what I can do on a BASH command line, and not to find non-bash solutions. I tend to perform a lot of complicated tasks from the command line and want another tool in the toolbox.

Meanwhile, here's the function that handles getting the next line from the file. As you can see, it modifies an on-disk file each time it reads/removes a line. That's what seems hackish, but I'm not coming up with anything better, since FIFO's didn't work w/o setvbuf() in bash.

#
# Get/remove the first line from FILE, using LOCK as a semaphore (with
# short sleep for collisions).  Returns the text on standard output,
# returns zero on success, non-zero when file is empty.
#
parallel__nextLine() 
{
  local line rest file=$1 lock=$2

  # Wait for lock...
  until ln "${file}" "${lock}" 2>/dev/null
  do sleep 1
     [ -s "${file}" ] || return $?
  done

  # Open, read one "line" save "rest" back to the file:
  exec 3<"$file"
  read line <&3 ; rest=$(cat<&3)
  exec 3<&-

  # After last line, make sure file is empty:
  ( [ -z "$rest" ] || echo "$rest" ) > "${file}"

  # Remove lock and 'return' the line read:
  rm -f "${lock}"
  [ -n "$line" ] && echo "$line"
}

解决方案

#adjust these as required
args_per_proc=1 #1 is fine for long running tasks
procs_in_parallel=4

xargs -n$args_per_proc -P$procs_in_parallel povray < list

Note the nproc command coming soon to coreutils will auto determine the number of available processing units which can then be passed to -P

这篇关于多线程bash编程 - 广义的方法？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！