是否可以识别 TLS 信息.在请求响应中? [英] Is it possible to identify TLS info. in requests response?

问题描述

我正在使用 python 的请求模块.我可以得到服务器的响应头和应用层数据:

I am using python's requests module. I can get the server's response headers and application layer data as:

import requests
r = requests.get('https://yahoo.com')
print(r.url)  

我的问题:请求是否允许检索传输层数据(服务器的 TLS 选择版本、密码套件等?).

My question: Does requests allow retrieving Transport layer data (the server's TLS selected version, ciphersuite, etc. ?).

推荐答案

这是一个有效的快速丑陋的猴子补丁版本:

Here is a quick ugly monkey patching version that works:

import requests
from requests.packages.urllib3.connection import VerifiedHTTPSConnection

SOCK = None

_orig_connect = requests.packages.urllib3.connection.VerifiedHTTPSConnection.connect

def _connect(self):
    global SOCK
    _orig_connect(self)
    SOCK = self.sock

requests.packages.urllib3.connection.VerifiedHTTPSConnection.connect = _connect

requests.get('https://yahoo.com')
tlscon = SOCK.connection
print 'Cipher is %s/%s' % (tlscon.get_cipher_name(), tlscon.get_cipher_version())
print 'Remote certificates: %s' % (tlscon.get_peer_certificate())
print 'Protocol version: %s' % tlscon.get_protocol_version_name()

这产生:

Cipher is ECDHE-RSA-AES128-GCM-SHA256/TLSv1.2
Remote certificates: <OpenSSL.crypto.X509 object at 0x10c60e310>
Protocol version: TLSv1.2

然而，这很糟糕，因为猴子修补并依赖于唯一的全局变量，这也意味着您无法检查重定向步骤中发生的情况，等等.

However it is bad because monkey patching and relying on a unique global variable, which also means you can not inspect what happens at redirect steps, and so on.

也许可以将一些工作变成 Transport Adapter，以获取底层连接作为请求的属性(可能是会话或其他内容).但这可能会造成泄漏，因为在当前实现中，底层套接字会尽快被丢弃(参见 使用Python请求时如何获取底层socket).

Maybe with some work that can be turned out as a Transport Adapter, to get the underlying connection as a property of the request (probably of the session or something). That may create leaks though, because in the current implementation the underlying socket is thrown away as quickly as possible (see How to get the underlying socket when using Python requests).

这有效，并且符合框架(没有全局变量，应该处理重定向等.虽然可能有一些代理可以做，比如为 proxy_manager_for 添加一个覆盖)，但是这是更多的代码.

This works, and is in line with the framework (no global variable, should handle redirects, etc. there may something to do for proxies though, like adding an override for proxy_manager_for too), but it is a lot more code.

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.connectionpool import HTTPSConnectionPool
from requests.packages.urllib3.poolmanager import PoolManager


class InspectedHTTPSConnectionPool(HTTPSConnectionPool):
    @property
    def inspector(self):
        return self._inspector

    @inspector.setter
    def inspector(self, inspector):
        self._inspector = inspector

    def _validate_conn(self, conn):
        r = super(InspectedHTTPSConnectionPool, self)._validate_conn(conn)
        if self.inspector:
            self.inspector(self.host, self.port, conn)

        return r


class InspectedPoolManager(PoolManager):
    @property
    def inspector(self):
        return self._inspector

    @inspector.setter
    def inspector(self, inspector):
        self._inspector = inspector

    def _new_pool(self, scheme, host, port):
        if scheme != 'https':
            return super(InspectedPoolManager, self)._new_pool(scheme, host, port)

        kwargs = self.connection_pool_kw
        if scheme == 'http':
            kwargs = self.connection_pool_kw.copy()
            for kw in SSL_KEYWORDS:
                kwargs.pop(kw, None)

        pool = InspectedHTTPSConnectionPool(host, port, **kwargs)
        pool.inspector = self.inspector
        return pool


class TLSInspectorAdapter(HTTPAdapter):
    def __init__(self, inspector):
        self._inspector = inspector
        super(TLSInspectorAdapter, self).__init__()

    def init_poolmanager(self, connections, maxsize, block=False, **pool_kwargs):
        self.poolmanager = InspectedPoolManager(num_pools=connections, maxsize=maxsize, block=block, strict=True, **pool_kwargs)
        self.poolmanager.inspector = self._inspector


def connection_inspector(host, port, connection):
    print 'host is %s' % host
    print 'port is %s' % port
    print 'connection is %s' % connection
    sock = connection.sock
    sock_connection = sock.connection
    print 'socket is %s' % sock
    print 'Protocol version: %s' % sock_connection.get_protocol_version_name()
    print 'Cipher is %s/%s' % (sock_connection.get_cipher_name(), sock_connection.get_cipher_version())
    print 'Remote certificate: %s' % sock.getpeercert()



url = 'https://yahoo.com'
s = requests.Session()
s.mount(url, TLSInspectorAdapter(connection_inspector))
r = s.get(url)

是的，socket 和 connection 之间的命名有很多混淆:请求使用了一个具有一组连接的连接池"，实际上它们是，对于 HTTPS，一个 PyOpenSSL WrappedSocket，它本身有一个底层真正的 TLS 连接(即 PyOpenSSL Connection 对象).因此，connection_inspector 中出现了奇怪的形式.

Yes, there is a lot of confusion in naming between socket and connection: requests uses a "connection pool" that has a set of connections, which are in fact, for HTTPS, a PyOpenSSL WrappedSocket, which has itself an underlying real TLS connection (that is a PyOpenSSL Connection object). Hence the strange forms in connection_inspector.

但这会返回预期:

host is yahoo.com
port is 443
connection is <requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x10bb372d0>
socket is <requests.packages.urllib3.contrib.pyopenssl.WrappedSocket object at 0x10bb37410>
Protocol version: TLSv1.2
Cipher is ECDHE-RSA-AES128-GCM-SHA256/TLSv1.2
Remote certificate: {'subjectAltName': [('DNS', '*.www.yahoo.com'), ('DNS', 'add.my.yahoo.com'), ('DNS', '*.amp.yimg.com'), ('DNS', 'au.yahoo.com'), ('DNS', 'be.yahoo.com'), ('DNS', 'br.yahoo.com'), ('DNS', 'ca.my.yahoo.com'), ('DNS', 'ca.rogers.yahoo.com'), ('DNS', 'ca.yahoo.com'), ('DNS', 'ddl.fp.yahoo.com'), ('DNS', 'de.yahoo.com'), ('DNS', 'en-maktoob.yahoo.com'), ('DNS', 'espanol.yahoo.com'), ('DNS', 'es.yahoo.com'), ('DNS', 'fr-be.yahoo.com'), ('DNS', 'fr-ca.rogers.yahoo.com'), ('DNS', 'frontier.yahoo.com'), ('DNS', 'fr.yahoo.com'), ('DNS', 'gr.yahoo.com'), ('DNS', 'hk.yahoo.com'), ('DNS', 'hsrd.yahoo.com'), ('DNS', 'ideanetsetter.yahoo.com'), ('DNS', 'id.yahoo.com'), ('DNS', 'ie.yahoo.com'), ('DNS', 'in.yahoo.com'), ('DNS', 'it.yahoo.com'), ('DNS', 'maktoob.yahoo.com'), ('DNS', 'malaysia.yahoo.com'), ('DNS', 'mbp.yimg.com'), ('DNS', 'my.yahoo.com'), ('DNS', 'nz.yahoo.com'), ('DNS', 'ph.yahoo.com'), ('DNS', 'qc.yahoo.com'), ('DNS', 'ro.yahoo.com'), ('DNS', 'se.yahoo.com'), ('DNS', 'sg.yahoo.com'), ('DNS', 'tw.yahoo.com'), ('DNS', 'uk.yahoo.com'), ('DNS', 'us.yahoo.com'), ('DNS', 'verizon.yahoo.com'), ('DNS', 'vn.yahoo.com'), ('DNS', 'www.yahoo.com'), ('DNS', 'yahoo.com'), ('DNS', 'za.yahoo.com')], 'subject': ((('commonName', u'*.www.yahoo.com'),),)}

其他想法:

1. 如果您像 https://stackoverflow.com/a/22253656/6368697 那样进行猴子修补，您可能会删除大量代码 基本上是 poolmanager.pool_classes_by_scheme['http'] = MyHTTPConnectionPool;但这仍然是猴子修补，遗憾的是 PoolManager 没有为 pool_classes_by_scheme 变量提供一个很好的 API 以便能够轻松覆盖它
2. PyOpenSSL ssl_context 可能能够保持回调，这些回调将在 TLS 握手期间调用并获取底层数据；然后在 init_poolmanager 中，您只需要在调用超类之前在 kwargs 中设置 ssl_context；https://gist.github.com/aiguofer/1eb881ccf199d4aaa2097d;aaa2097d;aaa2097d;a可能不是因为实际上结构来自 ssl.create_default_context 并且 ssl 远不如 PyOpenSSL 强大，我认为无法使用添加回调ssl，它们存在于 PyOpenSSL 中.万岁.

1. You may remove a lot of code if you do monkey patching like in https://stackoverflow.com/a/22253656/6368697 with basically poolmanager.pool_classes_by_scheme['http'] = MyHTTPConnectionPool; but this is still monkey patching, and it is sad that PoolManager does not give a nice API for the pool_classes_by_scheme variable to be able to easily override it
2. PyOpenSSL ssl_context may be able to hold callbacks that will be called during the TLS handshake and get the underlying data; then in init_poolmanager you would just need to setup the ssl_context in kwargs before calling superclass; this example in https://gist.github.com/aiguofer/1eb881ccf199d4aaa2097d87f93ace6a <= or maybe not because in fact the structure comes from ssl.create_default_context and ssl is far less powerful than PyOpenSSL and I see no way to add callbacks using ssl, where they exist for PyOpenSSL. YMMV.

附注:

1. 一旦你发现你有 _validate_conn ，你可以在它获得正确的连接对象时覆盖它，生活就会更轻松
2. 特别是如果你正确地在顶部进行导入，你需要使用分布在请求中的 urllib3 包，而不是真正的"urllib3 否则你会得到很多奇怪的错误，因为两者中的相同方法都没有有相同的签名...

1. Once you find out you have _validate_conn that you can override as it gets the proper connection object, life is easier
2. And especially if you do the import on top correctly, you need to use the urllib3 packages that are distributed inside requests, not the "real" urllib3 otherwise you get a lot of strange errors, because the same methods in both do not have the same signatures...

这篇关于是否可以识别 TLS 信息.在请求响应中?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！