Node.js批量下载和Bluebird承诺 [英] Node.js bulk download and Bluebird promises

问题描述

我正在为node.js编写一个批量下载器，并试图了解bluebird的承诺. 我想限制并行请求和磁盘写入的数量. 据我了解，Promise.map()和{concurrent: }应该可以满足我的要求.

I'm writing a bulk downloader for node.js and trying to understand bluebird promises. I want to limit the number of parallel requests and disk writes. As I understand it, Promise.map() with {concurrent: } should do what I want.

因为pipe()和http.get()不能自动实现，所以我尝试使用自定义的Promise.

Because pipe() and http.get() can't automatically be promisified, I'm trying to use custom promises.

但是我不完全了解then()机制. 对我来说，听起来应该只在整个链条都实现后才兑现返回的承诺.

But I don't fully understand the then() mechanism. To me, it sounds like the returned promise should only be fulfilled when the whole chain has been fulfilled.

但是，在我的代码中，map()似乎只等待链中的第一个承诺，并且许多请求和磁盘写入是并行发生的.

However, in my code only the first promise in the chain appears to be waited for by map(), and many request and disk writes happen in parallel.

import Promise from 'bluebird';
import fs from 'fs';
import https from 'https';

Promise.promisifyAll(fs);
Promise.map(Images, image => {
        console.log("Opening image " + image.id);
        let file = fs.createWriteStream(dir + '/' + image.id + '.jpg');
        return new Promise((resolve, reject) => {
              console.log("Downloading image " + image.id);
              https.get(image.url, resolve).on("error", reject);
          })
          .then(response => {
              response.pipe(file);
              console.log("Saving image " + image.id);
              return new Promise((resolve, reject) => {
                  file.on("finish", resolve);
                  file.on("error", reject);
              });
          })
          .then(() => {
              console.log("Finished writing image " + image.id);
              file.close();
          })
          .catch(e => {
              console.log("Error during image save of " + image.id + ": " + e.code)
          });
    }, {concurrent: 50})
      .then(res => {
          console.log("Finished writing all images")
      })
      .catch(e => {
          console.log("Some images failed to be written: " + e.code)
      });
}

我做错了什么?您能帮助我了解承诺履行和拒绝的流程吗?

What am I doing wrong? Can you help me understand the flow of promise fulfillment and rejection?

推荐答案

据我了解，您正在尝试下载Promise的多个图像.实际上，您不需要传播fs.您应该使用request模块以方便下载.

From what I can understand, you're trying to download multiple images with promise. Actually you don't need to promisify fs. You should use request module for easier downloading.

这是我想出的最短的工作示例

Here's a shortest working example I can come up with

var Promise = require('bluebird');
var path = require('path');
var fs = require('fs');
var request = require('request');

var images = [{
    url: 'http://bluebirdjs.com/img/logo.png',
    file_name: 'bluebird.png'
}, {
    url: 'http://design.ubuntu.com/wp-content/uploads/ubuntu-logo32.png',
    file_name: 'ubuntu.png'
}, {
    url: 'https://www.raspberrypi.org/wp-content/uploads/2012/03/raspberry-pi-logo.png',
    file_name: 'raspberry-pi.png'
}];

// To Download Serially
Promise.each(images, image => new Promise((resolve, reject) => {
    console.log('Downloading Image: ' + image.file_name);
    request(image.url).on('error', reject).pipe(fs.createWriteStream(path.join(__dirname, image.file_name))).on('finish', () => {
        console.log('Downloaded Image: ' + image.file_name);
        resolve();
    });
})).then(() => {
    console.log('All Image Downloaded!');
}).catch(err => {
    console.error('Failed: ' + err.message);
});

// To Download in Parallel (with 2 maximum concurrent jobs)
Promise.map(images, image => new Promise((resolve, reject) => {
    console.log('Downloading Image: ' + image.file_name);
    request(image.url).on('error', reject).pipe(fs.createWriteStream(path.join(__dirname, image.file_name))).on('finish', () => {
        console.log('Downloaded Image: ' + image.file_name);
        resolve();
    });
}), {
    concurrency: 2
}).then(() => {
    console.log('All Image Downloaded!');
}).catch(err => {
    console.error('Failed: ' + err.message);
});

这篇关于Node.js批量下载和Bluebird承诺的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！