如何在 Python 中可靠地处理网络数据 [英] How to reliably process web-data in Python

问题描述

我正在使用以下代码从网站获取数据:

I'm using the following code to get data from a website:

time_out = 4

def tryconnect(turl, timer=time_out, retries=10):
    urlopener = None
    sitefound = 1
    tried = 0
    while (sitefound != 0) and tried < retries:
        try:
            urlopener = urllib2.urlopen(turl, None, timer)
            sitefound = 0
        except urllib2.URLError:
            tried += 1
    if urlopener: return urlopener
    else: return None

[...]

urlopener = tryconnect('www.example.com')
if not urlopener:
    return None
try:
    for line in urlopener:
        do stuff
except httplib.IncompleteRead:
    print 'incomplete'
    return None
except socket.timeout:
    print 'socket'
    return None
return stuff

有没有一种方法可以处理所有这些异常，而不必每次都编写大量样板代码?

Is there a way I can handle all these exceptions without having so much boilerplate code everytime?

谢谢！

推荐答案

你也可以避免第一个函数中的一些样板代码:

You can avoid some boilerplate code in the first function too:

time_out = 4

def tryconnect(turl, timer=time_out, retries=10):
    for tried in xrange(retries):
        try:
            return urllib2.urlopen(turl, None, timer)
        except urllib2.URLError:
            pass
    return None

第二个:

urlopener = tryconnect('www.example.com')
if urlopener:
    try:
        for line in urlopener:
            do stuff
    except (httplib.IncompleteRead, socket.timeout), e:
        print e
        return None
else:
    return None

这篇关于如何在 Python 中可靠地处理网络数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！