仅使用 python 请求下载文档的一部分 [英] Only download a part of the document using python requests
问题描述
我正在使用 python-requests 编写一个网络爬虫.
I'm writing a web scraper using python-requests.
每个页面都超过 1MB，但我需要提取的实际数据在文档流的早期阶段，所以我浪费时间下载很多不必要的数据.
Each page is over 1MB, but the actual data I need to extract is very early on in the document's flow, so I'm wasting time downloading a lot of unnecessary data.
如果可能，我想在文档的源代码中出现所需数据时立即停止下载，以节省时间.
If possible I would like to stop the download as soon as the required data appears in the document's source code, in order to save time.
比如我只想提取abc"Div中的文字，其余的文档没用:
For example, I only want to extract the text in the "abc" Div, the rest of the document is useless:
<html>
<head>
<title>My site</title>
</head>
<body>
<div id="abc">blah blah...</div>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Mauris fermentum molestie ligula, a pharetra eros mollis ut.</p>
<p>Quisque auctor volutpat lobortis. Vestibulum pellentesque lacus sapien, quis vulputate enim mollis a. Vestibulum ultrices fermentum urna ac sodales.</p>
<p>Nunc sit amet augue at dolor fermentum ultrices. Curabitur faucibus porttitor vehicula. Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>
<p>Etiam sed leo at ipsum blandit dignissim ut a est.</p>
</body>
</html>
目前我只是在做:
r = requests.get(URL)
推荐答案
你想在这里使用的叫做 Range HTTP Header.
What you want to use here is called Range HTTP Header.
参见:http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html(特别是范围上的位).
另见关于自定义标头的API文档
示例:
from requests import get
url = "http://download.thinkbroadband.com/5MB.zip"
headers = {"Range": "bytes=0-100"} # first 100 bytes
r = get(url, headers=headers)
这篇关于仅使用 python 请求下载文档的一部分的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
