Node.js:如何限制请求列表? [英] Node.js: How to throttle a list of requests?

问题描述

我正在编写一个node.js应用，该应用需要从提供商的页面列表中获取一些数据:

 变量列表= [{url:'http://www.example.com/1'}，{url:'http://www.example.com/2'}，...{url:'http://www.example.com/N'}，]; 

目前，我正在使用 async.each ，效果很好:

  async.each(列表，//第一个参数是项目数组function(elem，callback){//第二个参数是将每个项目传递给的函数request(elem.url，function(error，response，body){if(！error&& response.statusCode == 200){console.log(body);}})，}，function(err){//第三参数是在完成所有操作后要调用的函数如果(错误){console.error('最终异步回调中的错误:'，err);}}); 

唯一的问题是，由于来自同一IP的请求过多，站点的服务器有时(可以理解)以 403 ( forbidden )状态码进行响应在时间单位...

我看到 async 也提供了 whilst()方法，其示例是:

  var count = 0;async.whilst(函数(){返回计数<5;}，功能(回调){数++;setTimeout(callback，1000);}，函数(err){//5秒过去了}); 

但是我没有看到如何使用它与列表一起使用，或者如何结合使用 async.each ...:-(

所以答案是:如何限制(限制)node.js中的异步请求列表?

PS:更清楚地说，我不希望(如果可能)排队请求，因为一个请求可能需要很长时间才能完成..:我只希望以定义的时间间隔(例如，每个请求之间5到10秒...)启动请求.

---

更新:

在alireza david评论之后，我确实尝试使用async.eachLimit，对我来说，这看起来非常有前途...这是它的用法示例，在模块github 解决方案

大多数情况下403表示您应该限制请求，因为Web服务器认为您正在进行DDOS攻击.

在这种情况下，您应该 async.eachLimit()

  async.eachLimit(obj.files，1000，功能(文件，完整){完全的();}，函数(err){}); 

更新我想知道了， limit 选项是并发请求的数量.您应该减少这个数字(我的意见是2或3，仅用于测试)

I'm writing a node.js app which needs to get some data from a list of pages from a provider:

var list = [
  { url: 'http://www.example.com/1' },
  { url: 'http://www.example.com/2' },
  ...
  { url: 'http://www.example.com/N' },
];

Currently I'm using async.each, which works nicely:

async.each(
  list, // 1st param is the array of items
  function(elem, callback) { // 2nd param is the function that each item is passed to
    request(elem.url, function (error, response, body) {
      if (!error && response.statusCode == 200) {
        console.log(body);
      }
    }),
  },
  function(err) { // 3rd param is the function to call when everything's done
    if (err) {
      console.error('Error in the final async callback:', err);
    }
  }
);

The only problem is that the site's server some times (understandably) responds with a 403 (forbidden) status code, due to an excess of requests from the same IP in the time unit...

I see async provides a whilst() method, too, whose example is:

var count = 0;
async.whilst(
  function () { return count < 5; },
  function (callback) {
    count++;
    setTimeout(callback, 1000);
  },
  function (err) {
    // 5 seconds have passed
  }
);

But I don't see how to use it to use it with a list, or how to use it combined with async.each... :-(

So the answer is: How do I limit (throttle) a list of async requests in node.js?

P.S.: To be clearer, I don't want (if possible) to queue the requests, since a request could possibly take a long time to complete...: I just want the requests to be initiated at defined temporal intervals (say 5 ~ 10 seconds between each request...).

---

UPDATE:

After alireza david comment, I did try using async.eachLimit, which looked very promising, to me... This is an example of it's usage, on the module github site:

async.eachLimit(
    obj.files,
    limit
    function (file, complete) {
      complete();
    },
    function (err) {
    }
);

But the limit usage is not documented, and it's not clear to me... If anybody has any clue...

解决方案

Most of the time 403 means you should limit your requests, Because web server thinks you doing DDOS attack.

In this situation you should async.eachLimit()

async.eachLimit(obj.files, 1000,
    function (file, complete) {
      complete();
    },
    function (err) {

    });

UPDATE I think got it, The limit options is number of concurrence requests. You should decrease this number (My opinion is 2 or 3 just for test)

这篇关于Node.js:如何限制请求列表?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！