Python 请求不处理仅来自一台机器的丢失的中间证书 [英] Python Requests not handling missing intermediate certificate only from one machine

问题描述

我正在开发一个运行 CentOS (Linux) 的机器，在尝试访问特定子域进行工作时遇到以下错误:

Traceback(最近一次调用最后一次):... # 我的代码，相关调用是 requests.get(url)文件/usr/local/lib/python2.7/site-packages/requests/api.py"，第 60 行，在 get返回请求('get', url, **kwargs)文件/usr/local/lib/python2.7/site-packages/requests/api.py"，第 49 行，在请求中返回 session.request(method=method, url=url, **kwargs)请求中的文件/usr/local/lib/python2.7/site-packages/requests/sessions.py"，第 457 行resp = self.send(prep, **send_kwargs)文件/usr/local/lib/python2.7/site-packages/requests/sessions.py"，第 569 行，在发送中r = adapter.send(request, **kwargs)发送中的文件/usr/local/lib/python2.7/site-packages/requests/adapters.py"，第 420 行引发 SSLError(e, request=request)requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

根据https://www.digicert.com/help/，子域是没有发送所需的中间证书"(这是 DigiCert 发现的唯一问题).但是，当我从 Mac 笔记本电脑运行代码时，我的代码可以毫无问题地处理这个问题，Chrome 和 Safari 也是如此.我在笔记本电脑和 linux 机器上都运行 Python 2.7.5.我在 linux 机器上运行请求 1.2.0，在我的笔记本电脑上运行 2.2.1，但我将两者都升级到 2.4.3，它们仍然没有相同的行为.

也可能相关 - 相同的证书正在用于发送中间证书的其他一些子域，并且我的笔记本电脑和 linux 机器都没有这些问题，所以不应该是我的笔记本电脑有linux盒子没有的root CA.

有谁知道为什么它在我的 linux 机器上不起作用以及我该如何解决它?

解决方案

我花了一天的时间来彻底理解和解决这个问题，所以我认为很高兴与大家分享我的发现:-)！这是我的结果:

提供不完整的证书链是 SSL 服务器配置中的一个常见缺陷，通常会省略中间证书.例如，我正在使用的一个网站不包含常见的 DigiCert中级"证书.证书DigiCert TLS RSA SHA256 2020 CA1"在服务器的响应中.

由于这种配置缺陷很常见，大多数但不是所有现代浏览器都实施了一种称为AIA Fetching"的技术.即时修复此问题(参见例如 https://www.thesslstore.com/blog/aia-fetching/).

Python 的 SSL 支持不支持 AIA Fetching，并且依赖于来自服务器的完整证书链；否则会抛出异常，像这样

SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] 证书验证失败:无法获取本地颁发者证书 (_ssl.c:1124)')))

关于是否应将 AIA Fetching 添加到 Python 的讨论正在进行中，例如在这个线程中:https://bugs.python.org/issue18617#msg293894.p>

我的印象是，在可预见的未来，这仍将是一个悬而未决的问题.

现在，我们该如何解决这个问题?

1. 安装certifi，如果你还没有这样做，或者更新它

pip 安装证书

或

pip install certifi --upgrade

许多(但不是全部)Python 模块可以使用来自 certifi 的证书，并且 certifi 从 Mozilla CA 证书倡议 获取它们(https://wiki.mozilla.org/CA).基本上，certifi 从 Mozilla 站点创建一个干净的 *.pem 文件，并提供一个轻量级的 Python 接口来访问该文件.

1. 将丢失的证书下载为 PEM 语法中的文件，例如来自 https://www.digicert.com/kb/digicert-root-certificates.htm，或来自受信任的浏览器.

2. 找到 certifi *.PEM 证书文件

    导入证书打印(certifi.where())

注意:我建议先激活虚拟环境(例如conda activate )你要使用证书. 文件路径会有所不同.如果您将此应用到您的基础环境，任何潜在的有缺陷的证书都会使您的所有代码的整个 SSL 机制处于危险之中.

示例:/Users/用户名/anaconda3/envs/environment_name/lib/python3.8/site-packages/certifi/cacert.pem

使用一个简单的文本编辑器，打开该文件，然后在标题之后的开头插入缺少的证书，就像这样

<上一页>#### CA 根证书包##...-----开始证书-----+I2tIJLYrVJmuzHZ9bjPvXj1hJeRPG/cUJ9WIQDgLGBAfr5yjK7tI4nhyfFK3TUqNaX3sNk+crOU6J---> 这是附加证书.+I2tIJLYrVJmuzHZ9bjPvXj1hJeRPG/cUJ9WIQDgLGBAfr5yjK7tI4nhyfFK3TUqNaX3sNk+crOU6J-----结束证书-----

包括开始和结束标记很重要.

保存文件，一切准备就绪！

您可以使用以下几行测试它是否有效:

<上一页># Python 3导入 urllib.request 导入证书导入请求URL = 'https://www.the_url_that_caused_the_trouble.org'print('正在尝试 urllib.request.urlopen().')r = urllib.request.urlopen(URL)print(f'urllib.request.urlopen ================ {r.read()[:80]}')print('尝试 requests.get().')r = requests.get(URL)print(f'requests.get() ================ {r.text[:80]}')

注意:一般的 SSL 证书，例如对于 openssl，可能位于其他地方，因此您可能必须在那里尝试相同的方法:

/Users/用户名/anaconda3/envs/environment_name/ssl

瞧！

注意事项:

1. 当您更新 certifi 或创建新的虚拟环境时，更改可能会丢失，但我认为这实际上是一个好的设计，因为它不会对您的整个系统进行临时的安全调整.
2. 当然，下载证书的过程存在潜在的安全风险 - 如果下载被盗，您的整个 SSL 链也可能被盗.
3. certifi 的维护滞后于 Mozilla 发布的证书.如果您想通过 certifi 使用最新版本的 Mozilla CA 捆绑包，您可以使用来自的我的脚本 https://github.com/mfhepp/update_certifi_certificates.

I'm working on a box that's running CentOS (Linux), and I'm running into the following error when try to access a particular subdomain for work:

Traceback (most recent call last):
  ... # My code, relevant call is requests.get(url)
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 60, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 420, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

According to https://www.digicert.com/help/, the subdomain "is not sending the required intermediate certificate" (and that's the only problem DigiCert found). However, my code handles this without problem when I run it from my Mac laptop, and so do both Chrome and Safari. I'm running Python 2.7.5 on both my laptop and the linux box. I was running requests 1.2.0 on the linux box and 2.2.1 on my laptop, but I upgraded both to 2.4.3 and they still don't have the same behavior.

Also possibly relevant - the same certificate is being used with some other subdomains where the intermediate certificate is being sent, and neither my laptop nor the linux box has any problems with those, so it shouldn't be that my laptop has a root CA that the linux box doesn't have.

Does anyone know why it isn't working from my linux box and how I can fix it?

解决方案

I spent a day to understand and fix this issue completely, so I thought it will be nice to share my findings with everybody :-)! Here are my results:

It is a common flaw in SSL server configurations to provide an incomplete chain of certificates, often omitting intermediate certificates. For instance, a site I was working with did not include the common DigiCert "intermediate" certificate "DigiCert TLS RSA SHA256 2020 CA1" in the server's response.

Because this configuration flaw is common, most but not all modern browsers implement a technique called "AIA Fetching" to fix this on the fly (see e.g. https://www.thesslstore.com/blog/aia-fetching/).

Python's SSL support does not support AIA Fetching and depends on a complete chain of certificates from the server; otherwise it throws an exception, like so

SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1124)')))

There is an ongoing discussion about whether AIA Fetching should be added to Python, e.g. in this thread: https://bugs.python.org/issue18617#msg293894.

My impression is that this will remain an open issue for the foreseeable future.

Now, how can we fix that?

1. Install certifi, if you have not done so, or update it

pip install certifi

or

pip install certifi --upgrade

Many (but not all) Python modules can use the certificates from certifi, and certifi takes them from the Mozilla CA Certificate initiative (https://wiki.mozilla.org/CA). Basically, certifi creates a clean *.pem file from the Mozilla site and provides a lightweight Python interface for accessing that file.

2. Download the missing certificate as a file in PEM syntax, e.g. from https://www.digicert.com/kb/digicert-root-certificates.htm, or from a trusted browser.

3. Locate the certifi *.PEM certificate file with

    import certifi
    print(certifi.where())
   

Note: I recommend to first activate the virtual environment (e.g. conda activate <envname>) you want to use the certificate with. The file path will differ. If you apply this to your base environment, any potential flawed certificate will put the entire SSL mechanism for all your code at risk.

Example: /Users/username/anaconda3/envs/environment_name/lib/python3.8/site-packages/certifi/cacert.pem

Take a simple text editor, open that file, and insert the missing certificate at the beginning right after the header, like so

##
## Bundle of CA Root Certificates
##
...
-----BEGIN CERTIFICATE-----
+I2tIJLYrVJmuzHZ9bjPvXj1hJeRPG/cUJ9WIQDgLGB
Afr5yjK7tI4nhyfFK3TUqNaX3sNk+crOU6J
---> This is the additional certificate.
+I2tIJLYrVJmuzHZ9bjPvXj1hJeRPG/cUJ9WIQDgLGB
Afr5yjK7tI4nhyfFK3TUqNaX3sNk+crOU6J
-----END CERTIFICATE-----

It is important to include the begin and end markers.

Save the file and you should be all set!.

You can test that it works with the following few lines:

# Python 3
import urllib.request import certifi import requests

 
URL = 'https://www.the_url_that_caused_the_trouble.org' 
print('Trying urllib.request.urlopen().') 
r = urllib.request.urlopen(URL)
print(f'urllib.request.urlopen
================
 {r.read()[:80]}')
print('Trying requests.get().') 
r = requests.get(URL)
print(f'requests.get()
================
 {r.text[:80]}')

Note: The general SSL certificates, e.g. for openssl, might be located elsewhere, so you may have to try the same approach there:

/Users/username/anaconda3/envs/environment_name/ssl

Voila!

Notes:

1. When you update certifi or create a new virtual environment, the changes will likely be lost, but I think that is actually good design, because it does not perpetuate a temporary security tweak to your entire system.
2. Naturally, the process of downloading the certificate is a potential security risk - if that download is compromised, your entire SSL chain might be, too.
3. The maintenance of certifi lags behind the Mozilla releases of certificates. If you want to use the most current version of the Mozilla CA bundles with certifi, you can use my script from https://github.com/mfhepp/update_certifi_certificates.

这篇关于Python 请求不处理仅来自一台机器的丢失的中间证书的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！