如何处理 IncompleteRead:在 python 中 [英] How to handle IncompleteRead: in python
问题描述
我正在尝试从网站获取一些数据.但是它返回我未完成阅读
.我试图获取的数据是一组巨大的嵌套链接.我在网上做了一些研究,发现这可能是由于服务器错误(A chunked transfer encoding finish before达到预期大小).我还在此链接上找到了上述解决方法
I am trying to fetch some data from a website. However it returns me incomplete read
. The data I am trying to get is a huge set of nested links. I did some research online and found that this might be due to a server error (A chunked transfer encoding finishing before
reaching the expected size). I also found a workaround for above on this link
但是,我不确定如何在我的情况下使用它.以下是我正在处理的代码
However, I am not sure as to how to use this for my case. Following is the code I am working on
br = mechanize.Browser()
br.addheaders = [('User-agent', 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1;Trident/5.0)')]
urls = "http://shop.o2.co.uk/mobile_phones/Pay_Monthly/smartphone/all_brands"
page = urllib2.urlopen(urls).read()
soup = BeautifulSoup(page)
links = soup.findAll('img',url=True)
for tag in links:
name = tag['alt']
tag['url'] = urlparse.urljoin(urls, tag['url'])
r = br.open(tag['url'])
page_child = br.response().read()
soup_child = BeautifulSoup(page_child)
contracts = [tag_c['value']for tag_c in soup_child.findAll('input', {"name": "tariff-duration"})]
data_usage = [tag_c['value']for tag_c in soup_child.findAll('input', {"name": "allowance"})]
print contracts
print data_usage
请帮我解决这个问题.谢谢
Please help me with this.Thanks
推荐答案
link 您在问题中包含的只是一个执行 urllib 的 read() 函数的包装器,它会为您捕获任何不完整的读取异常.如果你不想实现整个补丁,你总是可以在你阅读链接的地方加入一个 try/catch 循环.例如:
The link you included in your question is simply a wrapper that executes urllib's read() function, which catches any incomplete read exceptions for you. If you don't want to implement this entire patch, you could always just throw in a try/catch loop where you read your links. For example:
try:
page = urllib2.urlopen(urls).read()
except httplib.IncompleteRead, e:
page = e.partial
对于python3
try:
page = request.urlopen(urls).read()
except (http.client.IncompleteRead) as e:
page = e.partial
这篇关于如何处理 IncompleteRead:在 python 中的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!