python请求从浏览器或urllib返回不同的网页 [英] python requests return a different web page from browser or urllib

问题描述

我使用请求来抓取某些内容的网页.
当我使用

I use requests to scrape webpage for some content.
When I use

import requests  
requests.get('example.org')

我得到的页面与我使用浏览器或使用

I get a different page from the one I get when I use my broswer or using

import urllib.request
urllib.request.urlopen('example.org')

我尝试使用 urllib 但它真的很慢.
在比较测试中，我做了它比 requests 慢 50% !!

I tried using urllib but it was really slow.
In a comparison test I did it was 50% slower than requests !!

你是如何解决这个问题的??

How Do you solve this??

推荐答案

经过大量调查，我发现该站点仅在附加到该站点的第一个访问者的标头中传递 cookie.

After a lot of investigations I found that the site passes a cookie in the header attached to the first visitor to the site only.

所以解决方案是使用 head 请求获取 cookie，然后使用 get 请求重新发送它们

so the solution is to get the cookies with head request, then resend them with your get request

import requests  
# get the cookies with head(), this doesn't get the body so it's FAST
cookies = requests.head('example.com')
# send get request with the cookies
result = requests.get('example.com', cookies=cookies)

现在它比 urllib + 相同的结果更快 :)

Now It's faster than urllib + the same result :)

这篇关于python请求从浏览器或urllib返回不同的网页的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！