我应该为批量异步 GET 请求使用多个 HttpClients 吗? [英] Should I utilize multiple HttpClients for bulk async GET requests?

问题描述

我有一个场景，我需要在尽可能短的时间内发出大量 GET 请求(想想大约 1000 个).

I have a scenario where I need to make a large number of GET requests in as little time as possible (think around 1000).

我知道通常最好保留一个客户端并尽可能多地重复使用它:

I know generally it's best to keep a single client and reuse it as much as possible:

// Create Single HTTP Client
HttpClient client = new HttpClient();

// Create all tasks
for (int x = 0; x < 1000; x++)
{
    tasks.Add(ProcessURLAsync($"https://someapi.com/request/{x}", client, x));
}

// wait for all tasks to complete.
Task.WaitAll(tasks.ToArray());

...

static async Task<string> ProcessURLAsync(string url, HttpClient client, int x)
{
    var response = await client.GetStringAsync(url);

    ParseResponse(response.Result, x);

    return response;
}

但这样做需要大约 70 秒才能完成所有请求.

But doing so takes approximately 70 seconds for all requests to complete.

另一方面，如果我预先创建多个客户端并在它们之间分发请求，则需要大约 3 秒才能完成:

On the other hand, If I create multiple clients beforehand and distribute the requests across them, the it takes about 3 seconds to complete:

// Create arbitrary number of clients
while (clients.Count < maxClients)
{
    clients.Add(new HttpClient());
}

// Create all tasks
for (int x = 0; x < 1000; x++)
{
    tasks.Add(ProcessURLAsync(
        $"https://someapi.com/request/{x}", clients[x % maxClients], x));
}

// Same same code as above

由于所请求数据的性质，我需要保持结果顺序或传递与请求关联的索引.

Due to the nature of the data requested, I need to either keep the results sequential or pass along the index associated with the request.

假设无法更改 API 以更好地格式化请求的数据，并且所有请求必须在继续之前完成，这个解决方案是明智的还是我错过了更聪明的替代方案?

Assuming the API cannot be changed to better format the requested data, and the all requests must complete before moving on, is this solution wise or am I missing a smarter alternative?

(为了简洁起见，我使用了任意数量的 HttpClient 而我将创建一个 HttpClient 池，一旦它收到响应就释放一个客户端，并且仅在没有空闲时创建一个新的)

(For the sake of brevity I've used an arbitrary number of HttpClient whereas I would create a pool of HttpClient that releases a client once it receives a response and only create a new one when none are free)

推荐答案

我建议两个主要更改.

1. 删除等待，以便可以同时进行多次下载时间.
2. 设置 DefaultConnectionLimit 为更大的数字(例如 50).

1. Remove the await so that multiple downloads can occur at the same time.
2. Set DefaultConnectionLimit to a larger number (e.g. 50).

这篇关于我应该为批量异步 GET 请求使用多个 HttpClients 吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！