从 Linux 上的命令队列进行并行处理(bash、python、ruby ......无论如何) [英] Parallel processing from a command queue on Linux (bash, python, ruby... whatever)

问题描述

我有一个包含 200 个命令的列表/队列，我需要在 Linux 服务器上的 shell 中运行这些命令.

I have a list/queue of 200 commands that I need to run in a shell on a Linux server.

我只想一次最多运行 10 个进程(从队列中).有些过程需要几秒钟才能完成，而另一些过程则需要更长的时间.

I only want to have a maximum of 10 processes running (from the queue) at once. Some processes will take a few seconds to complete, other processes will take much longer.

当一个进程完成时，我希望下一个命令从队列中弹出"并执行.

When a process finishes I want the next command to be "popped" from the queue and executed.

有没有人有解决这个问题的代码?

Does anyone have code to solve this problem?

进一步阐述:

在某种队列中，有 200 件工作需要完成.我希望最多同时进行 10 件工作.当一个线程完成一项工作时，它应该向队列询问下一项工作.如果队列中没有更多工作，线程应该死亡.当所有的线程都死了就意味着所有的工作都已经完成了.

There's 200 pieces of work that need to be done, in a queue of some sort. I want to have at most 10 pieces of work going on at once. When a thread finishes a piece of work it should ask the queue for the next piece of work. If there's no more work in the queue, the thread should die. When all the threads have died it means all the work has been done.

我试图解决的实际问题是使用 imapsync 将 200 个邮箱从旧邮件服务器同步到新邮件服务器.有些用户的邮箱很大，需要很长时间才能同步，有些用户的邮箱很小，同步很快.

The actual problem I'm trying to solve is using imapsync to synchronize 200 mailboxes from an old mail server to a new mail server. Some users have large mailboxes and take a long time tto sync, others have very small mailboxes and sync quickly.

推荐答案

我想你可以使用 make 和 make -j xx 命令来做到这一点.

I would imagine you could do this using make and the make -j xx command.

也许是这样的makefile

Perhaps a makefile like this

all : usera userb userc....

usera:
       imapsync usera
userb:
       imapsync userb
....

make -j 10 -f 生成文件

make -j 10 -f makefile

这篇关于从 Linux 上的命令队列进行并行处理(bash、python、ruby ......无论如何)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！