使用请求的响应时间极长 [英] Extremely long response time using requests

问题描述

说明

我有一个运行python应用程序的AWS ec2实例(ubuntu 16).在其中，我调用了一些Facebook Account Kit API以及Google Play商店API.在我两周前重新启动实例之前，它们都工作正常.

I have an AWS ec2 instance (ubuntu 16) that runs a Python application. In which I call some Facebook Account Kit APIs and also Google Play Store APIs. They all work perfectly fine until I reboot the instance two weeks before.

重新启动后，请求将花费5分钟以上的时间才能完成，这完全是不可接受的.我必须手动将超时设置为超过10分钟，以使请求能够完成.

After the reboot, the requests take more than 5 mins to finish, which is totally not acceptable. I have to manually set the timeout to over 10mins in order to let the request to be finished.

问题仅发生在我的一台服务器上，我在另一台服务器上以相同的环境运行，它的响应时间不到一秒，效果很好.

The problem only occurs on one of my servers, I run with the same environment on another server, it works perfectly fine with the response time of less than a second.

临时修复

要暂时解决此问题，然后使用代理服务器完成请求.

To temporarily fix the issue, I then use a proxy server to finish the request.

1. API服务器(服务器出现超时问题)
2. 代理服务器运行python脚本并返回结果
3. API服务器(服务器出现超时问题)将响应返回给客户端

情况

1. 我尝试在API服务器上使用curl，它的响应时间还不到1秒.
2. 我使用 requests 在python环境中进行了尝试，响应时间非常糟糕，超过5分钟.

1. I tried using curl on the API server, it also has the response time of less than 1 second.
2. I tried on the python environment using requests, and the response time is terrible, above 5 mins.

1. 如果我设置标头 {'Connection':'keep-alive'} ，则第二个请求将是正常的.
2. 我打开了日志记录，似乎请求停留在建立到目标的连接上.

1. If I set the header {'Connection' : 'keep-alive' }, the second request onwards would be normal.
2. I turned on the logging, and it seems the request stuck on establishing connection to the destination.

我尝试了自己编写的API，超过5分钟的响应时间也很糟糕.

当前代码

响应时间较慢的请求.

url_get_access_token = "https://graph.accountkit.com/v1.3/access_token?grant_type=authorization_code&code=%s&access_token=AA|%s|%s"
url_get_access_token = url_get_access_token % (token, self.facebook_app_id, self.facebook_account_kit_scert)
response = requests.get(url_get_access_token)
body = response.json()

我上面提到的代理服务器是同一子网中的另一个实例，但是我使用DNS服务器进行呼叫.

The proxy server I mentioned above is another instance in the same subnet, but I call with a DNS server.

response = requests.get("https://proxyserver.com/somepath", params={})

由于它仅影响其中一台服务器，这是DNS问题还是AWS配置?请帮忙，谢谢.

As it only affects one of the servers, would that be a DNS problem or AWS config? Please help, thank you.

更新

定时卷发的结果，似乎使用iPv6进行调用会花费更多时间.

Result of timed curls, it seems that call with iPv6 takes a lot more time.

$ time curl -4 -s https://graph.accountkit.com/v1.3
$ time curl -6 -s https://graph.accountkit.com/v1.3

iPv4

real    0m0.665s
user    0m0.068s
sys 0m0.020s

iPv6

real    2m7.180s
user    0m0.008s
sys 0m0.000s

推荐答案

我想到了两个项目.

调试:

$ cat /etc/resolv.conf

$ time dig aaaa graph.accountkit.com

您可能列出了多个名称服务器，并不是所有人都反应灵敏，这样一来，您就会因为在死者身上超时而遭受漫长的查找.

It is possible that you have multiple nameservers listed, and not all of them are responsive, so you suffer from long lookups as it times out on the dead one(s).

调试:

$ time curl -4 -s https://graph.accountkit.com/v1.3
$ time curl -6 -s https://graph.accountkit.com/v1.3

它将说"Invalid OAuth 2.0 Access Token"(无效的OAuth 2.0访问令牌)，是的，很好.我们感兴趣的是连接需要多长时间，发送GET并检索Web文档.

It will say "Invalid OAuth 2.0 Access Token", yeah, yeah, that's fine. What we're interested in is how long it takes to connect, send the GET, and retrieve the web document.

此域同时提供A和AAAA地址.如果要烘烤IPv6，则可能需要一段时间使 requests.get()可以故障转移到IPv4.

This domain offers both an A and an AAAA address. If IPv6 transport is toast it could take a while for requests.get() to failover to IPv4.

编辑

有人破坏了您的IPv6传输.在现代互联网中这是不可接受的.丢包超时可能导致经过127秒的时间.诸如 traceroute6 和 ping6 之类的工具可以帮助您或网络专业人员进行诊断损失在哪里.可能是ACL太紧，正在丢弃不应该的IPv6数据包.丢弃ICMP尤其糟糕.为了使TCP正常工作，必须交付ICMP.

Someone broke your IPv6 transport. That's not acceptable in the modern internet. Dropped packet timeouts likely led to the 127-second elapsed time. Tools like traceroute6 and ping6 can help you or a network professional diagnose where the lossage is. Possibly an ACL is too tight, is discarding IPv6 packets that it shouldn't. Discarding ICMPs would be especially bad. For TCP to work correctly, ICMPs must be delivered.

tcpdump (或Wireshark)数据包跟踪会有所帮助确切地确定向南走的是什么.您可能正在遭受 PMTUD黑洞.看看是否显示任何"packet too too"(数据包太大)，ICMP报告:

A tcpdump (or Wireshark) packet trace would help identify exactly what went south. It is possible you are suffering from PMTUD black-holing. See if this displays any "packet too big" ICMP reports:

$ sudo tcpdump -tvvvni eth0 icmp6 and ip6[40+0]==2 

仅查看出站端口443 TCP重新传输的时间将为为什么失败两分钟提供很多启示然后比特突然开始流动.

Just looking at the timing of outbound port 443 TCP retrans would shed a lot of light on why things fail for two minutes and then the bits suddenly start flowing.

这篇关于使用请求的响应时间极长的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！