使用多个进程队列推出的bash工作 [英] Queue using several processes to launch bash jobs
问题描述
我需要运行许多（数百）的shell命令，但我只希望有一个最大的一次（从队列）上运行4个过程。每个进程将持续数个小时。
I need to run many (hundreds) commands in shell, but I only want to have a maximum of 4 processes running (from the queue) at once. Each process will last several hours.
当一个过程完成后，我想从队列中的下一个命令是弹出和执行。
When a process finishes I want the next command to be "popped" from the queue and executed.
我也希望能够开始后添加更多的进程，这将是巨大的，如果我可以从队列中删除一些工作，或者至少是空的队列中。
I also want to be able to add more process after the beginning, and it will be great if I could remove some jobs from the queue, or at least empty the queue.
我所看到的，利用makefile的解决方案，但如果我有我所有的开始前的命令列表这只是工作。使用mkfifo子sjobq，和其他人也尝试过，但我从来没有能达到我的要求...
I have seen solutions using makefile, but this only work if I have all my list of commands before the beginning. Also tried using mkfifo sjobq, and others, but I never could reach my needs...
有没有人有code来解决这个问题？
Does anyone have code to solve this problem?
编辑：在回应马克瑟特查
In response to Mark Setchell
与尾-f和并行解决方案几乎是完美的，但是当我做它，它始终保持不发动最后4个命令，直到我添加更多的，等等，我不知道为什么，这是相当麻烦...
The solution with tail -f and parallel is almost perfect, but when I do it, it always keep not launching the last 4 commands until I add more, and so on, I don't know why, and it is quite troublesome...
至于Redis的，很好的解决方案还可以，但它需要更多的时间来掌握它的全部。
As for Redis, good solution also, but it takes more time to master all of it.
谢谢！
推荐答案
使用GNU并行，使作业队列是这样的：
Use GNU Parallel to make a job queue like this:
# Clear out file containing job queue
> jobqueue
# Start GNU Parallel processing jobs from queue
# -k means "keep" output in order
# -j 4 means run 4 jobs at a time
tail -f jobqueue | parallel -k -j 4
# From another terminal, submit 40 jobs to the queue
for i in {1..40}; do echo "sleep 5;date +'%H:%M:%S Job $i'"; done >> jobqueue
另一种选择是使用Redis的 - 见我的答案在这里<一个href=\"http://stackoverflow.com/questions/22219262/run-several-jobs-parallelly-and-efficiently/22220082#22220082\">Run几项工作并行，高效地
这篇关于使用多个进程队列推出的bash工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
