Python 请求错误 10060 [英] Python requests error 10060
问题描述
我有一个抓取网站的脚本.直到今天它运行完美,但是现在不这样做了.
I have a script that crawls a website. Untill today it ran perfect, However it does not do so now.
它给我以下错误:
Connection Aborted Error(10060 ' A connection attempt failed becvause the connected party did not properly respond after a period of time, or established a connection failed because connected host has failed to respond'
我一直在寻找答案和设置,但我不知道如何解决这个问题...
I have been looking into answers ans settings but i cannot figure out how to fix this...
在 IE 中我没有使用任何代理(连接 -> 局域网设置 -> 代理 = 已禁用)
In IE i am not using any Proxy (Connection -> Lan Settings-> Proxy = Disabled)
它在这段代码中中断,有时是第一次运行,有时是第二次......等等
it breaks in this piece of code, somethimes the first run, somethimes the 2nd.. and so on
def geturls(functionurl, runtime):
startCrawl = requests.get(functionurl, headers=headers)
mainHtml = BeautifulSoup(startCrawl.content, 'html.parser')
mainItems = mainHtml.find("div",{"id": "js_multiselect_results"})
for tag in mainItems.findAll('a', href=True):
tag['href'] = urlparse.urljoin(url,tag['href'])
if shorturl in tag['href'] and tag['href'] not in visited:
if any(x in tag['href'] for x in keepout):
falseurls.append(tag['href'])
elif tag['href'] in urls:
doubleurls.append(tag['href'])
else:
urlfile.write(tag['href'] + "\n")
urls.append(tag['href'])
totalItemsStart = str(mainHtml.find("span",{"id": "sab_header_results_size"}))
if runtime == 1:
totalnumberofitems[0] = totalItemsStart
totalnumberofitems[0] = strip_tags(totalnumberofitems[0])
return totalnumberofitems
我该如何解决这个问题?
How can i fix this?
推荐答案
尝试增加 requests.get
方法的 timeout
参数:
Try increasing the timeout
parameter of your requests.get
method :
requests.get(functionurl, headers=headers, timeout=5)
但是很有可能您的脚本被服务器阻止以防止报废尝试.如果是这种情况,您可以尝试通过设置适当的标题来伪造 Web 浏览器.
But the odds are that your script is being blocked by the server to prevent scrapping attempts . If this is the case you can try faking a web browser by setting appropriate headers .
{"User-Agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 GTB7.1 (.NET CLR 3.5.30729)", "Referer": "http://example.com"}
这篇关于Python 请求错误 10060的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!