Node.js批处理 [英] Nodejs batch processing

问题描述

一些概念性问题

我有15个(例如)需要处理的文件.但是我不想一次处理它们.相反，我想开始处理其中5个文件(任何5个命令并不重要)，只要处理了这5个文件中的一个，另一个就可以开始了.这个想法是要同时处理最多5个文件，直到处理完所有文件为止.

I have 15 (for example) files that need to be processed. But i dont want to process them one at a time. Instead i want to start processing 5 of them (any 5 the order is not important) and as long one of these 5 files is processed another one to be started. The idea is to have max 5 files being processed at the same time until all files are processed.

试图在Node中解决这个问题，但总的来说我错过了如何实现的想法

Trying to work this out in Node but in general im missing the idea how this can be implemented

推荐答案

此类处理的更准确的名称可能是有限并行执行". Mario Casciaro在他的书《 Node.js设计模式》(第77页开始)中对此进行了很好的介绍.该模式的一个用例是，当您要控制一组可能导致过多负载的并行任务时.下面的例子来自他的书.

A more accurate name for this type of processing might be 'limited parallel execution'. Mario Casciaro covers this well in his book, Node.js Design Patterns beginning on page 77. One use case for this pattern is when you want to control a set of parallel tasks that could cause excessive load. The example below is from his book.

有限的并行执行模式

function TaskQueue(concurrency) {
  this.concurrency = concurrency;
  this.running = 0;
  this.queue = [];
}

TaskQueue.prototype.pushTask = function(task, callback) {
  this.queue.push(task);
  this.next();
}

TaskQueue.prototype.next = function() {
  var self = this;
  while(self.running < self.concurrency && self.queue.length) {
    var task = self.queue.shift();
    task(function(err) {
      self.running--;
      self.next();
    });
    self.running++;
  }
}

这篇关于Node.js批处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！