我应该为批量异步GET请求利用多个HttpClient吗? [英] Should I utilize multiple HttpClients for bulk async GET requests?

问题描述

我有一个场景，我需要在尽可能短的时间内发出大量的GET请求(大概1000个).

I have a scenario where I need to make a large number of GET requests in as little time as possible (think around 1000).

我知道通常最好保留一个客户端并尽可能地重用它:

I know generally it's best to keep a single client and reuse it as much as possible:

// Create Single HTTP Client
HttpClient client = new HttpClient();

// Create all tasks
for (int x = 0; x < 1000; x++)
{
    tasks.Add(ProcessURLAsync($"https://someapi.com/request/{x}", client, x));
}

// wait for all tasks to complete.
Task.WaitAll(tasks.ToArray());

...

static async Task<string> ProcessURLAsync(string url, HttpClient client, int x)
{
    var response = await client.GetStringAsync(url);

    ParseResponse(response.Result, x);

    return response;
}

但是，完成所有请求大约需要70秒.

But doing so takes approximately 70 seconds for all requests to complete.

另一方面，如果我事先创建了多个客户端并在其中分配请求，则大约需要3秒钟才能完成:

On the other hand, If I create multiple clients beforehand and distribute the requests across them, the it takes about 3 seconds to complete:

// Create arbitrary number of clients
while (clients.Count < maxClients)
{
    clients.Add(new HttpClient());
}

// Create all tasks
for (int x = 0; x < 1000; x++)
{
    tasks.Add(ProcessURLAsync(
        $"https://someapi.com/request/{x}", clients[x % maxClients], x));
}

// Same same code as above

由于所请求数据的性质，我需要使结果保持顺序或传递与请求关联的索引.

Due to the nature of the data requested, I need to either keep the results sequential or pass along the index associated with the request.

假设无法更改API以更好地格式化请求的数据，并且所有请求必须在继续之前完成，这是明智的解决方案，还是我缺少更聪明的选择?

Assuming the API cannot be changed to better format the requested data, and the all requests must complete before moving on, is this solution wise or am I missing a smarter alternative?

(为简洁起见，我使用了任意数量的HttpClient，而我将创建一个HttpClient池，该池在收到响应后将释放客户端，并且仅在没有空闲时创建一个新客户端)

(For the sake of brevity I've used an arbitrary number of HttpClient whereas I would create a pool of HttpClient that releases a client once it receives a response and only create a new one when none are free)

推荐答案

我建议两个主要更改.

1. 删除等待状态，以便可以同时进行多个下载 时间.
2. 设置将DefaultConnectionLimit 设置为更大的数字(例如50).

1. Remove the await so that multiple downloads can occur at the same time.
2. Set DefaultConnectionLimit to a larger number (e.g. 50).

这篇关于我应该为批量异步GET请求利用多个HttpClient吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！