Python Scrapy 301 重定向 [英] Python Scrapy 301 redirects
问题描述
在抓取给定网站时,我在打印重定向的 url(301 重定向后的新 URL)时遇到了一些问题.我的想法是只打印它们而不是刮它们.我目前的一段代码是:
I have a little problem in printing the redirected urls (new URLs after 301 redirection) when scraping a given website. My idea is to only print them and not scrape them. My current piece of code is:
import scrapy
import os
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
class MySpider(CrawlSpider):
name = 'rust'
allowed_domains = ['example.com']
start_urls = ['http://example.com']
rules = (
# Extract links matching 'category.php' (but not matching 'subsection.php')
# and follow links from them (since no callback means follow=True by default).
# Extract links matching 'item.php' and parse them with the spider's method parse_item
Rule(LinkExtractor(), callback='parse_item', follow=True),
)
def parse_item(self, response):
#if response.status == 301:
print response.url
但是,这不会打印重定向的 url.任何帮助将不胜感激.
However, this does not print the redirected urls. Any help will be appreciated.
谢谢.
推荐答案
要解析不是 200 的任何响应,您需要执行以下操作之一:
To parse any responses that are not 200 you'd need to do one of these things:
您可以在 settings.py
文件中设置设置 HTTPERROR_ALLOWED_CODES = [301,302,...]
.或者,如果您想为所有代码启用它,您可以改为设置 HTTPERROR_ALLOW_ALL = True
.
You can set setting HTTPERROR_ALLOWED_CODES = [301,302,...]
in settings.py
file. Or if you want to enable it for all codes you can set HTTPERROR_ALLOW_ALL = True
instead.
将 handle_httpstatus_list
参数添加到您的蜘蛛.在你的情况下是这样的:
Add handle_httpstatus_list
parameter to your spider. In your case something like:
class MySpider(scrapy.Spider):
handle_httpstatus_list = [301]
# or
handle_httpstatus_all = True
请求范围
您可以在请求中设置这些 meta
键 handle_httpstatus_list = [301, 302,...]
或 handle_httpstatus_all = True
:
scrapy.request('http://url.com', meta={'handle_httpstatus_list': [301]})
要了解更多信息,请参阅 HttpErrorMiddleware
To learn more see HttpErrorMiddleware
这篇关于Python Scrapy 301 重定向的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!