请求，机械化，urllib失败，但cURL工作 [英] Requests, Mechanize, urllib fails but cURL works

问题描述

在尝试通过请求访问此网站时，我会收到：

 （'Connection aborted。'，错误（54，'Connection reset by peer'）
  

我也试图通过mechanize和urllib访问网站，都失败了。

我尝试过 requests.get（）参数 verify = True ， stream = True ，我也试过一个带有cURL头的请求。

我试图移动到urllib / Mechanize作为替代，但两者都给出了相同的错误。

请求如下：

 导入请求
 import cookielib 
 
 url =https： //datamuster.marketdatasuite.com/Account/LogOn?ReturnUrl=%2fProfile%2fList
 
 header = {
'Accept'：'text / html，application / xhtml + xml，application /xml;q=0.9,image/webp,*/*;q=0.8'，
'Accept-Encoding'：'gzip，deflate，sdch'，
'Accept-Language'：'en -US，en; q = 0.8'，
'User-Agent'：'Mozilla / 5.0（Macintosh; Intel Mac OS X 10_10_0）AppleWebKit / 537.36（KHTML，like Gecko）Chrome / 38.0.2125.122 Safari / 537.36 '
} 
 
 jar = cookielib.CookieJar（）
s = requests.Session（）
 s.headers.update（header）
 
r = s.get（url，cookies = jar）
  

cURL测试带标题：

  $ curl -v -I -H....测试：Header ....https://datamuster.marketdatasuite.com / Account / LogOn？ReturnUrl =％2fProfile％2fList 
 
 *在DNS缓存中找不到主机名
 *尝试54.252.86.7 ... 
 *连接到datamuster.marketdatasuite。 com（54.252.86.7）port 443（＃0）
 * TLS 1.2使用TLS_RSA_WITH_AES_128_CBC_SHA256 
 *连接*服务器证书：datamuster.marketdatasuite.com 
 *服务器证书：COMODO SSL CA 
 *服务器证书：AddTrust外部CA根
> HEAD / Account / LogOn？ReturnUrl =％2fProfile％2fList HTTP / 1.1 
> User-Agent：curl / 7.37.1 
>主机：datamuster.marketdatasuite.com 
>接受：* / * 
> ....测试：标题.... 
> 
< HTTP / 1.1 200 OK 
  

解决方案

简单明了，所以我想我会发布一个新的答案，以方便他人跟随。

$ b

• pyOpenSSL


• ndg-httpsclient


• < a href =http://pyasn1.sourceforge.net/ =nofollow> pyasn1

但是，当安装 pip install pyOpenSSL 时，pyOpenSSL可能会导致问题。我实际上要删除我现有的openssl，因为pyOpenSSL版本0.14似乎不工作：

  pip uninstall pyOpenSSL 
  

以下命令安装了所有必要的依赖项：

  pip install pyOpenSSL == 0.13 ndg-httpsclient pyasn1 
  

请求现在使用SNI在python 2上工作。

---

继续阅读pyOpenSSL版本的问题。 0.14 ...

安装ver。 0.14我收到以下错误：

 命令/usr/local/opt/python/bin/python2.7 -cimport setup_ol​​s，tokenize; __ file __ ='/ private / var / folders / 04 / 3f_y5fw166v03k7b51j1tsl80000gn / T / pip_build_alex / cryptography / setup.py'; exec（compile（getattr（tokenize，'open'，open） .replace（'\r\\\
'，'\\\
'），__file__，'exec'））install --record / var / folders / 04 / 3f_y5fw166v03k7b51j1tsl80000gn / T / pip-7QR71B-record /在/ private / var / folders / 04 / 3f_y5fw166v03k7b51j1tsl80000gn / T / pip_build_alex / cryptography 
中存储故障的调试日志/用户/ alex / .pip / pip.log 
  

和pyOpenSSL安装为ver。 0.14不完整：

  $ pip show pyOpenSSL 
 --- 
名称：pyOpenSSL 
版本：0.14 
位置：/usr/local/lib/python2.7/site-packages 
需要：加密，六
  

$ requests.get（）尝试：

 导入请求
 response = requests.get（http://datamuster.marketdatasuite.com）
 
（...很多错误。 ..）
 raise ConnectionError（err，request = request）
 requests.exceptions.ConnectionError：（'Connection aborted。'，error（54，'Connection reset by peer'））
  

以下命令还原为pyOpenSSL ver。 0.13并更正问题

  pip uninstall pyOpenSSL 
 pip install pyOpenSSL == 0.13 
  
 
 
 
然后在python中：
 导入请求
 requests.get（http://datamuster.marketdatasuite.com）
 
< Response [200]> 
  
 
Whilst attempting to access this site through requests, I receive:
('Connection aborted.', error(54, 'Connection reset by peer'))
I have also tried to access the site through mechanize and urllib, both failed. However cURL works fine (see end for code).

I have tried requests.get() with combinations of parameters verify=True,stream=True and I have also tried a request with the cURL header. 

I tried to move to urllib / Mechanize as alternatives but both gave the same error.

My code for requests is as follows:
import requests
import cookielib

url = "https://datamuster.marketdatasuite.com/Account/LogOn?ReturnUrl=%2fProfile%2fList"

header = {
    'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Encoding':'gzip,deflate,sdch',
    'Accept-Language':'en-US,en;q=0.8',
    'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36'
}

jar = cookielib.CookieJar()
s = requests.Session()
s.headers.update(header)

r = s.get(url, cookies=jar)
cURL test with headers:
$ curl -v -I -H "....Testing: Header...." https://datamuster.marketdatasuite.com/Account/LogOn?ReturnUrl=%2fProfile%2fList

* Hostname was NOT found in DNS cache
*   Trying 54.252.86.7...
* Connected to datamuster.marketdatasuite.com (54.252.86.7) port 443 (#0)
* TLS 1.2 connection using TLS_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: datamuster.marketdatasuite.com
* Server certificate: COMODO SSL CA
* Server certificate: AddTrust External CA Root
> HEAD /Account/LogOn?ReturnUrl=%2fProfile%2fList HTTP/1.1
> User-Agent: curl/7.37.1
> Host: datamuster.marketdatasuite.com
> Accept: */*
> ....Testing: Header....
> 
< HTTP/1.1 200 OK

解决方案
This process wasn't totally straightforward so I thought I'd post a new answer to make it easy to follow for others.

Following this thread, I needed to install these libraries get SNI to work with Python 2:


pyOpenSSL
ndg-httpsclient
pyasn1


However, pyOpenSSL may cause problems when installed with pip install pyOpenSSL. I actually had to remove my existing openssl, since pyOpenSSL version 0.14 didn't seem to work:
pip uninstall pyOpenSSL
The following command installed all necessary dependencies:
pip install pyOpenSSL==0.13 ndg-httpsclient pyasn1
This should get requests to now work with SNI on python 2.



Keep reading for the issues with pyOpenSSL ver. 0.14...

When installing ver. 0.14 I get the following error:
Command /usr/local/opt/python/bin/python2.7 -c "import setuptools, tokenize;__file__='/private/var/folders/04/3f_y5fw166v03k7b51j1tsl80000gn/T/pip_build_alex/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/04/3f_y5fw166v03k7b51j1tsl80000gn/T/pip-7QR71B-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /private/var/folders/04/3f_y5fw166v03k7b51j1tsl80000gn/T/pip_build_alex/cryptography
Storing debug log for failure in /Users/alex/.pip/pip.log
and pyOpenSSL installs as ver. 0.14 incompletely:
$ pip show pyOpenSSL
---
Name: pyOpenSSL
Version: 0.14
Location: /usr/local/lib/python2.7/site-packages
Requires: cryptography, six
as can be seen from the requests.get() attempt:
import requests
response = requests.get("http://datamuster.marketdatasuite.com")

(...lots of errors...)
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(54, 'Connection reset by peer'))
The following commands revert to pyOpenSSL ver. 0.13 and correct the issue
pip uninstall pyOpenSSL
pip install pyOpenSSL==0.13
and then in python:
import requests
requests.get("http://datamuster.marketdatasuite.com")

<Response [200]>


                        
这篇关于请求，机械化，urllib失败，但cURL工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！