Python 3 Web Scraping 中的问题 HTTP 错误 403 [英] Problem HTTP error 403 in Python 3 Web Scraping

问题描述

我试图抓取一个网站进行练习，但我不断收到 HTTP 错误 403(它认为我是机器人吗)?

I was trying to scrape a website for practice, but I kept on getting the HTTP Error 403 (does it think I'm a bot)?

这是我的代码:

#import requests
import urllib.request
from bs4 import BeautifulSoup
#from urllib import urlopen
import re

webpage = urllib.request.urlopen('http://www.cmegroup.com/trading/products/#sortField=oi&sortAsc=false&venues=3&page=1&cleared=1&group=1').read
findrows = re.compile('<tr class="- banding(?:On|Off)>(.*?)</tr>')
findlink = re.compile('<a href =">(.*)</a>')

row_array = re.findall(findrows, webpage)
links = re.finall(findlink, webpate)

print(len(row_array))

iterator = []

我得到的错误是:

 File "C:\Python33\lib\urllib\request.py", line 160, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Python33\lib\urllib\request.py", line 479, in open
    response = meth(req, response)
  File "C:\Python33\lib\urllib\request.py", line 591, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Python33\lib\urllib\request.py", line 517, in error
    return self._call_chain(*args)
  File "C:\Python33\lib\urllib\request.py", line 451, in _call_chain
    result = func(*args)
  File "C:\Python33\lib\urllib\request.py", line 599, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

推荐答案

这可能是因为 mod_security 或某些类似的服务器安全功能会阻止已知的蜘蛛/机器人用户代理 (urllib 使用类似 python urllib/3.3.0 的东西，它很容易被检测到).尝试设置一个已知的浏览器用户代理:

This is probably because of mod_security or some similar server security feature which blocks known spider/bot user agents (urllib uses something like python urllib/3.3.0, it's easily detected). Try setting a known browser user agent with:

from urllib.request import Request, urlopen

req = Request('http://www.cmegroup.com/trading/products/#sortField=oi&sortAsc=false&venues=3&page=1&cleared=1&group=1', headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()

这对我有用.

顺便说一下，在您的代码中，您在 urlopen 行中的 .read 之后缺少 ()，但我认为它是打错字了.

By the way, in your code you are missing the () after .read in the urlopen line, but I think that it's a typo.

提示:由于这是练习，因此请选择其他非限制性站点.也许他们出于某种原因阻止了 urllib...

TIP: since this is exercise, choose a different, non restrictive site. Maybe they are blocking urllib for some reason...

这篇关于Python 3 Web Scraping 中的问题 HTTP 错误 403的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！