运行 1000 个请求，以便一次只运行 10 个 [英] Run 1000 requests so that only 10 runs at a time

问题描述

使用 node.js，我想以一次只能运行 10 个(或 n 个)的方式 http.get 多个远程 url.

如果本地发生异常(m 次)，我也想重试请求，但是当状态代码返回错误(5XX、4XX 等)时，请求被视为有效.

这对我来说真的很难理解.

问题:

1. 无法尝试捕获 http.get，因为它是异步的.
2. 需要一种在失败时重试请求的方法.
3. 我需要某种信号量来跟踪当前活动的请求计数.
4. 当所有请求完成后，我想获取列表中所有请求 url 和响应状态代码的列表，我想对其进行排序/分组/操作，因此我需要等待所有请求完成.

似乎对于每个异步问题都推荐使用 Promise，但我最终嵌套了太多 Promise 并且很快变得不可加密.

解决方案

有很多方法可以处理一次运行的 10 个请求.

1. 异步库 - 使用带有 .parallelLimit() 方法，您可以在其中指定要一次运行的请求数.

2. Bluebird Promise 库 - 使用 Bluebird Promise 库 和 request 库将您的 http.get() 包装成可以返回承诺的东西，然后使用 Promise.map()，并发选项设置为 10.

3. 手动编码 - 手动编码您的请求以启动 10 个，然后在每次完成时启动另一个.

在所有情况下，您都必须手动编写一些重试代码，并且与所有重试代码一样，您必须非常仔细地决定重试哪些类型的错误、重试的时间、重试尝试之间的退避程度当你最终放弃时(所有你没有指定的东西).

其他相关答案:

如何制作来自 nodejs 应用的数百万个并行 http 请求?

百万次请求，一次 10 个 - 手动编码示例><小时>

我的首选方法是使用 Bluebird 和 promise.按顺序包括重试和结果收集，可能看起来像这样:

const request = require('request');const Promise = require('bluebird');const get = Promise.promisify(request.get);让 remoteUrls = [...];//大的 URL 数组const maxRetryCnt = 3;const retryDelay = 500;Promise.map(remoteUrls, function(url) {让 retryCnt = 0;函数运行(){返回获取(网址).然后(功能(结果){//对这里的结果做任何你想做的事返回结果；}).catch(函数(错误){//决定你的重试策略是什么//在此处捕获所有错误，以便其他 URL 继续执行if (err 是重试类型 && retryCnt < maxRetryCnt) {++retryCnt;//稍等片刻后再试//链接到先前的 promise 上，因此 Promise.map() 仍然是//尊重我们的并发值返回 Promise.delay(retryDelay).then(run);}//如果没有重试成功，则使 value 为 null返回空；});}返回运行()；}, {concurrency: 10}).then(function(allResults) {//此处完成的所有操作和 allResults 包含的结果为 err URL 的 null});

With node.js I want to http.get a number of remote urls in a way that only 10 (or n) runs at a time.

I also want to retry a request if an exception occures locally (m times), but when the status code returns an error (5XX, 4XX, etc) the request counts as valid.

This is really hard for me to wrap my head around.

Problems:

1. Cannot try-catch http.get as it is async.
2. Need a way to retry a request on failure.
3. I need some kind of semaphore that keeps track of the currently active request count.
4. When all requests finished I want to get the list of all request urls and response status codes in a list which I want to sort/group/manipulate, so I need to wait for all requests to finish.

Seems like for every async problem using promises are recommended, but I end up nesting too many promises and it quickly becomes uncypherable.

解决方案

There are lots of ways to approach the 10 requests running at a time.

1. Async Library - Use the async library with the .parallelLimit() method where you can specify the number of requests you want running at one time.

2. Bluebird Promise Library - Use the Bluebird promise library and the request library to wrap your http.get() into something that can return a promise and then use Promise.map() with a concurrency option set to 10.

3. Manually coded - Code your requests manually to start up 10 and then each time one completes, start another one.

In all cases, you will have to manually write some retry code and as with all retry code, you will have to very carefully decide which types of errors you retry, how soon you retry them, how much you backoff between retry attempts and when you eventually give up (all things you have not specified).

Other related answers:

How to make millions of parallel http requests from nodejs app?

Million requests, 10 at a time - manually coded example

---

My preferred method is with Bluebird and promises. Including retry and result collection in order, that could look something like this:

const request = require('request');
const Promise = require('bluebird');
const get = Promise.promisify(request.get);

let remoteUrls = [...];    // large array of URLs

const maxRetryCnt = 3;
const retryDelay = 500;

Promise.map(remoteUrls, function(url) {
    let retryCnt = 0;
    function run() {
        return get(url).then(function(result) {
            // do whatever you want with the result here
            return result;
        }).catch(function(err) {
            // decide what your retry strategy is here
            // catch all errors here so other URLs continue to execute
            if (err is of retry type && retryCnt < maxRetryCnt) {
                ++retryCnt;
                // try again after a short delay
                // chain onto previous promise so Promise.map() is still
                // respecting our concurrency value
                return Promise.delay(retryDelay).then(run);
            }
            // make value be null if no retries succeeded
            return null;
        });
    }
    return run();
}, {concurrency: 10}).then(function(allResults) {
     // everything done here and allResults contains results with null for err URLs
});

这篇关于运行 1000 个请求，以便一次只运行 10 个的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！