请求库在HTTPS代理CONNECT上强制使用HTTP/1.1 [英] Requests Library Force Use of HTTP/1.1 On HTTPS Proxy CONNECT
问题描述
我的HTTP代理服务器行为异常.不幸的是，我无法控制代理服务器-它是IBM的企业"产品.代理服务器是用于软件测试的服务虚拟化解决方案的一部分.
基本问题(我认为*)是代理服务器发回HTTP/1.0响应.我可以从SOAP UI(一种Java应用程序)正常运行，并从命令行卷曲，但是Python拒绝连接.据我所知，Python运行正常，而其他两个运行不正常，因为服务器需要HTTP/1.1响应(它至少希望Host标头将服务请求路由到给定的存根).
是否有一种方法可以获取请求，或者底层的urllib3，或者甚至更低的http库，以始终使用http1.1，即使另一端似乎正在使用1.0?
这里有一个示例程序(不幸的是，它要求您安装具有RTCP的IBM Ration Integration Tester安装程序才能真正复制)以重现该问题:
将http.client导入为http_clienthttp_client.HTTPConnection.debuglevel = 1导入日志汇入要求logging.basicConfig()logging.getLogger().setLevel(logging.DEBUG)requests_log = logging.getLogger("requests.packages.urllib3")requests_log.setLevel(logging.DEBUG)requests_log.propagate = Truerequests.post("https://host:8443/axl"，headers = {"soapAction":'"CUCM:DB ver = 9.1 updateSipTrunk"'}，data ='< soapenv:信封xmlns:soapenv ="http://schemas.xmlsoap.org/soap/envelope/" xmlns:tns ="http://www.cisco.com/AXL/API/9.1">< soapenv:Header/>< soapenv:Body>< tns:updateSipTrunk><名称> PLACEHOLDER</名称>< newName> PLACEHOLDER</newName><目的地><目的地> p; 10.10.1.5</addressIpv4>< sortOrder> 1</sortOrder></destination>//目的地</tns:updateSipTrunk>//soapenv:Body></soapenv:Envelope，验证= False) (通过HTTPS_PROXY环境变量配置代理)
在错误之前调试输出，请注意HTTP/1.0:
INFO:requests.packages.urllib3.connectionpool:启动新的HTTPS连接(1):host.com发送:b'CONNECT host.com:8443 HTTP/1.0 \ r \ n'发送:b'\ r \ n'标头:主机:host.com:8443标头:Proxy-agent:Green Hat HTTPS Proxy/1.0 RHEL 6中出现的确切错误文本是:
requests.exceptions.SSLError:[SSL:SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3警报握手失败(_ssl.c:646) 即使在此处显示了Host标头，它也不会显示在网络上.我通过tcpdump确认了这一点:
14:03:14.315049 IP sourcehost.53214>desthost.com:标志[P.]，seq 0:32，ack 1，win 115，选项[nop，nop，TS val 2743933964 ecr 4116114841]，长度320x0000:0000 0c07 ac00 0050 56b5 4044 0800 4500 ....... PV.@ D..E.0x0010:0054 3404 4000 4006 2ca0 0af8 3f15 0afb .T4.@.@ .....0x0020:84f8 cfde 0c7f a4f8 280a 4ebd b425 8018 ........(.N ..％..0x0030:0073 da46 0000 0101 080a a38d 1c0c f556 .s.F ..... V0x0040:XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX ..CONNECT.host0x0050:XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX xx:8443.HTTP/1.00x0060:0d0a 当我用冗长的方式卷曲它时，输出结果如下:
*关于connect()到proxy proxy-host.com端口3199(#0)*正在尝试10.**.**.** ...已连接*连接到proxy-host.com(10.**.**.**)端口3199(#0)*建立到host.com的HTTP代理隧道:8443>CONNECT host.com:8443 HTTP/1.1>主持人:host.com:8443>用户代理:curl/7.19.7(x86_64-redhat-linux-gnu)libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2>代理连接:保持活动>soapAction:"CUCM:DB ver = 9.1 updateSipTrunk"><HTTP/1.0 200 OK<主持人:host.com:8443<代理:Green Hat HTTPS Proxy/1.0<*代理对连接请求的回答为确定"*使用证书路径初始化NSS:sql:/etc/pki/nssdb* CAfile:/path/to/store/ca-bundle.crtCApath:无*使用TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256的SSL连接 在此之后被截断.连接后，您可以从代理看到HTTP/1.0响应.curl的tcpdump也清楚地显示了主机头以及HTTP 1.1.
*我无法完全确定这是根本问题，因为我无法对其进行测试.我确实看到了HTTP/1.0响应，并且可以说我的非工作Python代码发送CONNECT HTTP/1.0消息，而工作的Java发送HTTP/1.1消息，Curl也是如此.这个问题可能无关紧要(尽管我发现不太可能)，或者Python的行为异常，而不是Java/curl.我只是不知道要知道什么.
那么，有没有一种方法可以强制urllib3/requests始终使用HTTP v1.1?
httplib ( Lib/httplib.py:788 :
def _tunnel(自己):self.send("CONNECT％s:％d HTTP/1.0 \ r \ n"％(self._tunnel_host，self._tunnel_port))对于标头，self._tunnel_headers.iteritems()中的值:self.send(％s:％s \ r \ n"％(标题，值))self.send("\ r \ n")< ...> 因此，除了通过编辑子例程之外，您不能在此处强制"它使用"HTTP/1.1".
如果代理不支持HTTP/1.0，则可能是问题所在-特别是1.0不需要 Host:标头，实际上，如您通过比较日志输出所看到的使用上面的代码， httplib 不会发送它.尽管实际上，代理可能会期望它.但是，在这种情况下，您应该从代理那里得到一个错误或响应CONNECT的东西-除非代理太乏味以至于它用 200 并尝试将God-knows-where连接到那里，这时您将超时.
您可以使 httplib 将 Host:头添加到CONNECT中，方法是将其添加到 _tunnel_headers 中(间接):
s = requests.Session()proxy_url = os.environ ['HTTPS_PROXY']s.proxies ["https"] = proxy_url#必须在此处指定代理，因为env变量仅由httplib代码检测到#while我们需要触发更早起作用的请求的代理逻辑#"https"表示任何https主机.由于会话保留cookie，#通过它向多个主机发出请求毫无意义.pm = s.get_adapter("https://").proxy_manager_for(proxy_url)pm.proxy_headers ['Host'] ="host.com"del pm，proxy_url< ...>s.get('https://host.com') I am having a problem with a misbehaving HTTP Proxy server. I have no control over the proxy server, unfortunately -- it's an 'enterprise' product from IBM. The proxy server is part of a service virtualization solution being leveraged for software testing.
The fundamental issue (I think*) is that the proxy server sends back HTTP/1.0 responses. I can get it to work fine from SOAP UI ( A Java application) and curl from the command line, but Python refuses to connect. From what I can tell, Python is behaving correctly, and the other two are not, as the server expects HTTP/1.1 responses (it wants Host headers, at the very least, to route the service request to a given stub).
Is there a way to get Requests, or the underlying urllib3, or the even farther down http lib to always use http1.1, even if the other end appears to be using 1.0?
Here is a sample program (unfortunately, it requires you to have an IBM Ration Integration Tester installation with RTCP to really replicate) to reproduce the problem:
import http.client as http_client
http_client.HTTPConnection.debuglevel = 1
import logging
import requests
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True
requests.post("https://host:8443/axl",
headers={"soapAction": '"CUCM:DB ver=9.1 updateSipTrunk"'},
data='<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:tns="http://www.cisco.com/AXL/API/9.1"><soapenv:Header/><soapenv:Body><tns:updateSipTrunk><name>PLACEHOLDER</name><newName>PLACEHOLDER</newName><destinations><destination><addressIpv4>10.10.1.5</addressIpv4><sortOrder>1</sortOrder></destination></destinations></tns:updateSipTrunk></soapenv:Body></soapenv:Envelope>',
verify=False)
(Proxy is configured via HTTPS_PROXY environment variable)
Debug output before the error, note the HTTP/1.0:
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): host.com
send: b'CONNECT host.com:8443 HTTP/1.0\r\n'
send: b'\r\n'
header: Host: host.com:8443
header: Proxy-agent: Green Hat HTTPS Proxy/1.0
The exact error text that occurs in RHEL 6 is:
requests.exceptions.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:646)
Even though the Host header is shown here, it does NOT show up on the wire. I confirmed this with a tcpdump:
14:03:14.315049 IP sourcehost.53214 > desthost.com: Flags [P.], seq 0:32, ack 1, win 115, options [nop,nop,TS val 2743933964 ecr 4116114841], length 32
0x0000: 0000 0c07 ac00 0050 56b5 4044 0800 4500 .......PV.@D..E.
0x0010: 0054 3404 4000 4006 2ca0 0af8 3f15 0afb .T4.@.@.,...?...
0x0020: 84f8 cfde 0c7f a4f8 280a 4ebd b425 8018 ........(.N..%..
0x0030: 0073 da46 0000 0101 080a a38d 1c0c f556 .s.F...........V
0x0040: XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX ..CONNECT.host
0x0050: XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX xx:8443.HTTP/1.0
0x0060: 0d0a
When I curl it with verbose, this is what the output looks like:
* About to connect() to proxy proxy-host.com port 3199 (#0)
* Trying 10.**.**.** ... connected
* Connected to proxy-host.com (10.**.**.**) port 3199 (#0)
* Establish HTTP proxy tunnel to host.com:8443
> CONNECT host.com:8443 HTTP/1.1
> Host: host.com:8443
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Proxy-Connection: Keep-Alive
> soapAction: "CUCM:DB ver=9.1 updateSipTrunk"
>
< HTTP/1.0 200 OK
< Host: host.com:8443
< Proxy-agent: Green Hat HTTPS Proxy/1.0
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /path/to/store/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
Truncated after this point. You can see the HTTP/1.0 response from the proxy after connecting. The curl's tcpdump also clearly shows the host header, as well as HTTP 1.1.
*I can't be entirely sure this is the fundamental issue, as I can't test it. I do see HTTP/1.0 responses, and can tell that my non-working Python code sends CONNECT HTTP/1.0 messages, while the working Java sends HTTP/1.1 messages, as does Curl. It's possible the problem is unrelated (although I find that unlikely) or that Python is misbehaving, and not Java/curl. I simply don't know enough to know for sure.
So, is there a way to force urllib3/requests to use HTTP v1.1 at all times?
httplib (which requests relies upon for HTTP(S) heavy lifting) always uses HTTP/1.0 with CONNECT:
def _tunnel(self):
self.send("CONNECT %s:%d HTTP/1.0\r\n" % (self._tunnel_host,
self._tunnel_port))
for header, value in self._tunnel_headers.iteritems():
self.send("%s: %s\r\n" % (header, value))
self.send("\r\n")
<...>
So you can't "force" it to use "HTTP/1.1" here other than by editing the subroutine.
This MAY be the problem if the proxy doesn't support HTTP/1.0 - in particular, 1.0 does not require a Host: header, and indeed, as you can see by comparing your log output with the code above, httplib does not send it. While, in verity, a proxy may expect it regardless. But if this is the case, you should've gotten an error from the proxy or something in response to CONNECT -- unless the proxy is so borken that it substitutes some default (or garbage) for Host:, returns 200 anyway and tries to connect God-knows-where, at which point you're getting timeouts.
You can make httplib add the Host: header to CONNECT by adding it to _tunnel_headers (indirectly):
s=requests.Session()
proxy_url=os.environ['HTTPS_PROXY']
s.proxies["https"]=proxy_url
# have to specify proxy here because env variable is only detected by httplib code
#while we need to trigger requests' proxy logic that acts earlier
# "https" means any https host. Since a Session persists cookies,
#it's meaningless to make requests to multiple hosts through it anyway.
pm=s.get_adapter("https://").proxy_manager_for(proxy_url)
pm.proxy_headers['Host']="host.com"
del pm,proxy_url
<...>
s.get('https://host.com')
这篇关于请求库在HTTPS代理CONNECT上强制使用HTTP/1.1的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
