无法解释的超时和延迟从 HttpClient SendAsync 请求获取响应 [英] Unexplained timeouts and delays on getting the response from HttpClient SendAsync requests

问题描述

我们有一个 .NET 4.7.2，它混合使用异步和同步代码(我理解这是禁忌).我们在 Windows 服务上使用 NancyFX.该服务获取休息调用并进行休息调用.线程池看起来很健康(整个进程只使用了 70 个线程).出于某种原因，一些 http 响应会延迟 10 秒，有时会延迟 100 秒并导致任务取消.

We have a .NET 4.7.2 that is using a mixture of asynchronous and synchronous code (i understand its a no-no). We are using NancyFX on a windows service. The service gets rest calls and makes rest calls. Thread pool seems healthy (whole process only using 70 threads). For some reason, some http responses are being delayed by 10s of seconds and sometimes by 100s of seconds and leading to a task cancelation.

代码结构如下

public async Task<Guid> SomeFunction()
{
   ...
   var response = await _httpClient.SendAsync(request, cancellationToken);
   ...
}

SomeFunction().Result

首先，我确信响应在网络某处由于某种原因被延迟了.但是我们已经排除了多种方法，最重要的是通过 perfview 查看 ETW 跟踪并看到数据包几乎立即返回(使用 Microsoft-Windows-NDIS-PacketCapture/PacketFragment)

First, I thought for sure the response was being delayed on the network somewhere for some reason. But we've ruled that out multiple ways, most significantly by looking at the ETW trace through perfview and seeing the packet make it back almost instantly (using Microsoft-Windows-NDIS-PacketCapture/PacketFragment)

其次，我确信这与异步方法上的 .Result 代码导致的线程池问题有关.但是，该进程上的线程再次稳定在 70 个线程.通过 perfview 我可以看到饥饿真的没有发生(使用 Microsoft-Windows-DotNETRuntime/ThreadPoolWorkerThreadAdjustment/Adjustment)

Secondly then, I was sure this had to do with threadpool issues caused by the .Result code on async methods. However, again, the threads on the process stay steady at 70 threads. Through perfview i can see that starvation is really not happening (using Microsoft-Windows-DotNETRuntime/ThreadPoolWorkerThreadAdjustment/Adjustment)

我也想过我可能会因为 await/async 和 .Result 陷入死锁状态，但死锁意味着请求永远不会完成，而不是它会延迟 10 秒.

I've also thought maybe I'm hitting into a deadlock situation with await/async and .Result, but a deadlock would imply that the request would never complete, not that it would be delayed by 10s of seconds.

我还仔细检查过我们只使用了一个 httpclient 实例，而且确实如此.

I've also double checked that we are only using one instance of httpclient and we indeed are.

还能是什么?

此时我们正在删除 .Result 并将其替换为适当的 async/await.但我没有证据表明这会解决问题，因为我没有看到任何死锁或线程耗尽的证据.

At this point we are removing the .Result and replacing it with proper async/await. But I have no proof that this will solve the issue since I'm not seeing any evidence of deadlocks or thread exhaustion.

这是一个perfview分析

Here is a perfview analysis

我们也在考虑关于我们正在以某种方式耗尽 http 连接的建议.我认为情况并非如此的一个原因是，根据 perfview，请求正在发出并且数据包将其返回，但响应并未构成 c# 堆栈.但是，这些性能计数器可能表明正在排队.

We're also looking at the suggestion that we are exhausting http connections in some capacity. One reason i don't think its the case is that, according to perfview, the request is being sent out and packets make it back, but the response just doesn't make it up the c# stack. However these performance counters might indicate that there is some queuing going on.

更新我们通过这个增加了http连接，它似乎生效了.

Update We have increased the http connections with this and it seems to take effect.

<connectionManagement>
  <add address="*" maxconnection="1024"/>
</connectionManagement>

上面显示的排队完全消失了.但是这些请求没有完成的问题仍然存在

The queuing shown above is completely gone. However the problem of these requests not completing remains

推荐答案

还能是什么?

您已经检查了我的第一个猜测，那就是线程池饥饿.

You've already checked my first guess, which would be thread pool starvation.

还有另一种可能性，具体取决于 API 调用的完成方式.如果对同一主机有许多并发请求，则 .NET 网络堆栈可能会限制您.非 ASP.NET 应用程序对同一主机的 2 个并发请求具有默认限制.在这种情况下，您有一个服务器应用程序，但没有一个 ASP.NET 应用程序，因此默认情况下您会进行相当严格的节流.

There's another possibility, depending on how the API calls are done. If there are many simultaneous requests to the same host, it is possible that the .NET networking stack may be throttling you. Non-ASP.NET apps have a default throttling of 2 simultaneous requests to the same host. In this case you have a server app but not an ASP.NET app, so you'd have that rather restrictive throttling on by default.

建议:将此代码放在您的启动中:

Recommendation: put this code in your startup:

ServicePointManager.DefaultConnectionLimit = int.MaxValue;

请注意，默认情况下 .NET Core 不会限制客户端 HTTP 请求，因此这只是模拟现代 .NET 平台上的默认行为.

Note that .NET Core does not throttle client HTTP requests by default, so this is just emulating the behavior which is the default on modern .NET platforms.

这篇关于无法解释的超时和延迟从 HttpClient SendAsync 请求获取响应的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！