HttpClient (C#) 在许多异步请求上失败? [英] HttpClient (C#) fails on many asynchronous requests?

问题描述

我正在使用 HttpClient 向外部 api 异步发出许多请求.我等待所有请求完成，然后使用其他代码中的响应.我的问题是，如果我发出太多请求，当我使用 Task.WhenAll 等待时，我的代码会抛出异常.

I'm using HttpClient to asynchronously make many requests to an external api. I wait for all the requests to complete, then use the responses in other code. My problem is that if I make too many requests, my code throws an exception when I use the Task.WhenAll to wait.

此代码最终将并行运行，我的意思是我将同时执行多组这些异步请求(即 10 组，每组 200 个异步请求).我已经实例化了一个 HttpClient，我正在使用 .NET 4.5 async/await 修饰符，如下所示:

This code will ultimately be run in parallel, by which I mean that I'll be doing many sets of these async requests at the same time (i.e. 10 sets of 200 async requests). I've instantiated an HttpClient that I'm using with .NET 4.5 async/await modifiers like this:

using (var client = new HttpClient())
{
    // make a list of tasks
    List<Task<HttpResponseMessage>> taskList;
    List<MyData> replies;
    for (int i = 0; i < MAX_NUMBER_REQUESTS; i++)
    {
        taskList.Add(client.GetAsync(externalUri);
    }

    List<HttpResponseMessage> responses = await Task.WhenAll(taskList);

    // read each response after they have returned
    foreach (var response in responses)
    {
        var reader = new System.IO.StreamReader(await response.Content.ReadAsStreamAsync());
        replies.Add(JsonConvert.DeserializeObject<MyData>(reader.ReadToEnd()));
        reader.Close();
} 

    foreach (var reply in replies)
    {
        // do something with the response from the external api call...
    }
}

我不断收到 TaskCanceledException.调查后，我发现这可能是超时问题.我不知道如何解决它.在使用 Task.WhenAll 和重复之前，我尝试以 100 个请求批处理我的请求，这很有效.然后当我与三组 100 个失败的请求并行运行时.我是否遗漏了什么，有人对此有任何见解吗?

I keep getting a TaskCanceledException. After looking into this I saw this is possibly a timeout issue. I have no idea how to fix it. I attempted to batch my requests at 100 requests before using the Task.WhenAll and repeating, which worked. Then when I ran that in parallel with three sets of 100 requests that fails. Am I missing something, does anyone have any insight into this?

推荐答案

调整 ServicePointManager.DefaultConnectionLimit.对于大量并发的请求，您可以将其设置为 int.MaxValue.

这篇关于HttpClient (C#) 在许多异步请求上失败?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！