从命令队列并行处理在Linux（bash中，从Python，Ruby ...什么） [英] Parallel processing from a command queue on Linux (bash, python, ruby... whatever)

问题描述

我有我需要在Linux服务器上的shell中运行200命令列表/队列。

I have a list/queue of 200 commands that I need to run in a shell on a Linux server.

我只希望有一个最大的一次（从队列）运行10个进程。一些进程将需要几秒钟即可完成，其他进程将需要更长的时间。

I only want to have a maximum of 10 processes running (from the queue) at once. Some processes will take a few seconds to complete, other processes will take much longer.

当一个过程完成后，我想从队列中的下一个命令是弹出和执行。

When a process finishes I want the next command to be "popped" from the queue and executed.

有没有人有code来解决这个问题？

Does anyone have code to solve this problem?

进一步阐述：

有200件作品需要做的，在某种队列。我想有工作最多10个一次回事。当一个线程完成一个作品便应要求队列为下一个作品。如果有队列中没有更多的工作，线程死了。当所有的线程已经死亡这意味着所有的工作已经完成。

There's 200 pieces of work that need to be done, in a queue of some sort. I want to have at most 10 pieces of work going on at once. When a thread finishes a piece of work it should ask the queue for the next piece of work. If there's no more work in the queue, the thread should die. When all the threads have died it means all the work has been done.

我试图解决实际问题是使用 imapsync 从旧的邮件服务器200邮箱同步到新的邮件服务器。有些用户具有较大的邮箱，并需要很长的时间TTO同步，人有非常小的邮箱和同步迅速。

The actual problem I'm trying to solve is using imapsync to synchronize 200 mailboxes from an old mail server to a new mail server. Some users have large mailboxes and take a long time tto sync, others have very small mailboxes and sync quickly.

推荐答案

我猜想，你可以用这个制作和化妆-j XX命令做的。

I would imagine you could do this using make and the make -j xx command.

也许这样的生成文件

all : usera userb userc....

usera:
       imapsync usera
userb:
       imapsync userb
....

请-j 10 -f生成文件

make -j 10 -f makefile

这篇关于从命令队列并行处理在Linux（bash中，从Python，Ruby ...什么）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！