运行1000个请求，以便一次只运行10个 [英] Run 1000 requests so that only 10 runs at a time

问题描述

使用node.js我希望 http.get 一些远程网址，一次只能运行10个（或n个）。

With node.js I want to http.get a number of remote urls in a way that only 10 (or n) runs at a time.

我还想在本地发生异常（m次）时重试请求，但是当状态代码返回错误（5XX，4XX等）时，请求计为有效。

I also want to retry a request if an exception occures locally (m times), but when the status code returns an error (5XX, 4XX, etc) the request counts as valid.

这对我来说真的很难理解。

This is really hard for me to wrap my head around.

问题：

1. 无法尝试捕获http.get，因为它是异步。


2. 需要一种方法在失败时重试请求。


3. 我需要某种信号量来跟踪当前活动的请求数。


4. 当所有请求都完成后我想获取所有请求URL的列表和列表中的响应状态代码我要排序/分组/操作，所以我需要等待所有请求完成。

1. Cannot try-catch http.get as it is async.
2. Need a way to retry a request on failure.
3. I need some kind of semaphore that keeps track of the currently active request count.
4. When all requests finished I want to get the list of all request urls and response status codes in a list which I want to sort/group/manipulate, so I need to wait for all requests to finish.

似乎建议使用promises的每个异步问题，但我最终嵌套太多的承诺，它很快变得无法修复。

Seems like for every async problem using promises are recommended, but I end up nesting too many promises and it quickly becomes uncypherable.

推荐答案

有很多方法可以同时处理10个请求。

There are lots of ways to approach the 10 requests running at a time.

1. 异步库 - 将async库与 .parallelLimit（）方法，您可以在其中指定一次要运行的请求数。

1. Async Library - Use the async library with the .parallelLimit() method where you can specify the number of requests you want running at one time.

Bluebird Promise Library - 使用 Bluebird promise library 和请求库包装你的 http.get（）到可以返回一个promise的东西，然后使用 Promise.map（），并发选项设置为 10 。

Bluebird Promise Library - Use the Bluebird promise library and the request library to wrap your http.get() into something that can return a promise and then use Promise.map() with a concurrency option set to 10.

手动编码 - 手动编码您的请求以启动10，然后每次完成，再启动另一个。

Manually coded - Code your requests manually to start up 10 and then each time one completes, start another one.

In在所有情况下，您将不得不手动编写一些重试代码，并且与所有重试代码一样，您将必须非常仔细地确定您重试的错误类型，重试时间，重试尝试之间的退回程度以及最终放弃（你没有指定的所有事情）。

In all cases, you will have to manually write some retry code and as with all retry code, you will have to very carefully decide which types of errors you retry, how soon you retry them, how much you backoff between retry attempts and when you eventually give up (all things you have not specified).

其他相关答案：

如何从nodejs app发出数百万条并行的http请求？

百万次请求，一次10个 - 手动编码示例

我首选的方法是使用Bluebird和promises。包括按顺序重试和结果收集，可能看起来像这样：

My preferred method is with Bluebird and promises. Including retry and result collection in order, that could look something like this:

const request = require('request');
const Promise = require('bluebird');
const get = Promise.promisify(request.get);

let remoteUrls = [...];    // large array of URLs

const maxRetryCnt = 3;
const retryDelay = 500;

Promise.map(remoteUrls, function(url) {
    let retryCnt = 0;
    function run() {
        return get(url).then(function(result) {
            // do whatever you want with the result here
            return result;
        }).catch(function(err) {
            // decide what your retry strategy is here
            // catch all errors here so other URLs continue to execute
            if (err is of retry type && retryCnt < maxRetryCnt) {
                ++retryCnt;
                // try again after a short delay
                // chain onto previous promise so Promise.map() is still
                // respecting our concurrency value
                return Promise.delay(retryDelay).then(run);
            }
            // make value be null if no retries succeeded
            return null;
        });
    }
    return run();
}, {concurrency: 10}).then(function(allResults) {
     // everything done here and allResults contains results with null for err URLs
});

这篇关于运行1000个请求，以便一次只运行10个的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！