Javascript（node.js）限制了子进程的数量 [英] Javascript (node.js) capped number of child processes

问题描述

希望我可以清楚地描述我正在寻找的东西。使用Node和Python。

hopefully I can describe what I'm looking for clearly enough. Working with Node and Python.

我正在尝试并行运行许多子进程（.py脚本，使用child_process.exec（）），但不再而不是一次指定的数字（比如2）。我批量收到未知数​​量的请求（比如这批请求有3个请求）。我想停止产生进程，直到其中一个结束。

I'm trying to run a number of child processes (.py scripts, using child_process.exec()) in parallel, but no more than a specified number at a time (say, 2). I receive an unknown number of requests in batches (say this batch has 3 requests). I'd like to stop spawning processes until one of the current ones finishes.

for (var i = 0; i < requests.length; i++) {

    //code that would ideally block execution for a moment
    while (active_pids.length == max_threads){
        console.log("Waiting for more threads...");
        sleep(100)
        continue
    };

    //code that needs to run if threads are available
    active_pids.push(i);

    cp.exec('python python-test.py '+ requests[i],function(err, stdout){
        console.log("Data processed for: " + stdout);

        active_pids.shift();

          if (err != null){
              console.log(err);
          }
    });
}

我知道虽然循环不起作用，但这是第一次尝试。

I know that while loop doesn't work, it was the first attempt.

我猜这有办法用

setTimeout(someSpawningFunction(){

    if (active_pids.length == max_threads){
        return
    } else {
        //spawn process?
    }

},100)

但我不能完全包裹我的脑袋它周围。

But I can't quite wrap my head around it.

或者

waitpid(-1)

在if语句中插入for循环代替while循环？但是我暂时无法安装waitpid（）模块。

Inserted in the for loop above in an if statement in place of the while loop? However I can't get the waitpid() module to install at the moment.

是的，我知道阻塞执行在JS中被认为非常糟糕，但在我的如果我需要它发生。如果可能的话，我宁愿避免使用外部集群管理器类型库。

And yes, I understand that blocking execution is considered very bad in JS, but in my case, I need it to happen. I'd rather avoid external cluster manager-type libraries if possible.

感谢您的帮助。

编辑/部分解决方案

丑陋的黑客将使用以下答案：这个SO问题（execSync（））。但这会阻止循环直到最后一个孩子完成。到目前为止，这是我的计划，但并不理想。

An ugly hack would be to use the answer from: this SO question (execSync()). But that would block the loop until the LAST child finished. That's my plan so far, but not ideal.

推荐答案

async 库的async＃timesLimitrel =nofollow> async.timesLimit 是完美的工具在这里使用。它允许您异步运行函数 n 次，但并行运行最多 k 的那些函数调用任何给定的时间。

async.timesLimit from the async library is the perfect tool to use here. It allows you to asynchronously run a function n times, but run a maximum of k of those function calls in parallel at any given time.

async.timesLimit(requests.length, max_threads, function(i, next){
    cp.exec('python python-test.py '+ requests[i], function(err, stdout){
        console.log("Data processed for: " + stdout);

        if (err != null){
            console.log(err);
        }

        // this task is resolved
        next(null, stdout);
    });
}, function(err, stdoutArray) {
  // this runs after all processes have run; what's next?
});

或者，如果您希望错误致命并停止循环，请致电 next（err，stdout）。

Or, if you want errors to be fatal and stop the loop, call next(err, stdout).

这篇关于Javascript（node.js）限制了子进程的数量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！