多个HTTP请求触发HTTP客户端超时 [英] Multiple HTTP requests trigger HTTP Client timeout

问题描述

我有一个应用程序可以异步发送500个HTTP请求.在15秒后 处理的所有请求均失败，原因是HTTP客户端 even 超时，此时请求的端点已返回200 OK.

I have an app that's sending 500 HTTP requests asynchronously. All requests processed after 15 seconds fail because of the timeout on the HTTP Client, even when the requested endpoint already returned a 200 OK.

代码非常简单.在这里，我们接收大量请求(500个)，并异步执行它们.应该注意的是，下面的功能是在基于消耗的计划上运行的Azure功能.

The code is very straight-forward. Here we take a chunk of requests (500), and execute them asynchronously. It should be noted that the function below is an Azure function running on the consumption-based plan.

    public async Task RunBatch(List<Request> requests)
    {
        if (requests != null && requests .Count > 0)
        {
            var tasks = new Task[requests.Count];
            var i = 0;
            foreach (var request in requests)
            {
                var request = new HttpRequestMessage(HttpMethod.Post, new Uri(request.Url));
                request.Content = new StringContent(request.BodyString, Encoding.UTF8, "application/json");
                tasks[i] = _httpClient.SendAsync(request);
                i++;
            }

            await Task.WhenAll(tasks);
        }
    }

以下代码存在于我的构造函数中

The following code exists in my constructor

_httpClient = new HttpClient();
_httpClient.Timeout = new TimeSpan(0, 0, 15); // 15 seconds

以下是来自Azure的日志.

Here are logs from Azure.

我希望每个请求的超时时间都为15秒.但是，每当我的服务器到处处理等待的请求时，我都需要它来给我一个准确的响应代码.这可能吗?

I'd like each request to have a timeout of 15 seconds. However, I need it to give me an accurate response code whenever my server gets around to processing the awaited request. Is this possible?

我应该注意:随着Http超时时间(1分钟)的延长，所有请求都会成功.

I should note: with a higher Http timeout (1 min), all requests succeed.

推荐答案

我个人认为，尝试发出500个并发请求总是容易出错.您提到您是异步进行的，但实际上，在启动500个热"任务然后等待所有任务完成之前，代码中并没有太多的异步性.

Personally, I think attempting to issue 500 concurrent requests is always going to be error prone. You mention that you're doing it asynchronously, but in reality there's not a whole lot of asynchrony in your code as you fire-up 500 "hot" tasks then wait for them all to finish.

我将使用信号量来控制一次可以发出多少个请求.您可能需要使用数字才能找到最佳位置.

I would use a semaphore to control how many requests can be made at once. You may have to play with the numbers to find the sweet spot.

以下代码在LINQPad中运行良好(尽管bing很快注意到了奇数请求，并开始向页面添加验证码):

The following code works well in LINQPad (although bing quickly notices the odd number of requests and starts adding a CAPTCHA to the page):

async Task Main()
{
    var httpClient = new HttpClient();
    var urls = Enumerable.Range(1, 500).Select(e => "https://www.bing.com/").ToList();

    // 10 concurrent requests - tweak this number
    var semaphore = new SemaphoreSlim(10, 10);

    var tasks = urls.Select(u => MakeRequest(u, semaphore, httpClient));

    var allResponses = await Task.WhenAll(tasks);

    // Do something with allResponses
}

private async Task<string> MakeRequest(string url, SemaphoreSlim semaphore, HttpClient httpClient)
{
    try
    {
        await semaphore.WaitAsync();
        var request = new HttpRequestMessage(HttpMethod.Get, new Uri(url));
        var response = await httpClient.SendAsync(request);

        // Add an optional delay for further throttling:
        //await Task.Delay(TimeSpan.FromMilliseconds(100));

        return await response.Content.ReadAsStringAsync();
    }
    finally
    {
        semaphore.Release();
    }
}

这篇关于多个HTTP请求触发HTTP客户端超时的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！