如何在scrapy中获取原始start_url(重定向前) [英] how to get the original start_url in scrapy (before redirect)

问题描述

我正在使用 Scrapy 抓取一些页面.我从 Excel 工作表中获取 start_urls，我需要将 url 保存在项目中.

I'm using Scrapy to crawl some pages. I fetch the start_urls from an excel sheet and I need to save the url in the item.

class abc_Spider(BaseSpider):
   name = 'abc'
   allowed_domains = ['abc.com']         
   wb = xlrd.open_workbook(path + '/somefile.xlsx')
   wb.sheet_names()
   sh = wb.sheet_by_name(u'Sheet1')
   first_column = sh.col_values(15)
   start_urls = first_column
   handle_httpstatus_list = [404]

   def parse(self, response):
      item = abcspiderItem()
      item['url'] = response.url

问题是该 url 被重定向到其他一些 url(因此在响应 url 中给出了其他内容).如何获取我从 excel 中得到的原始 url?

The problem is that the url gets redirected to some other url (and thus gives something else in the response url). How do I get the original url that I got from the excel?

推荐答案

您可以在 response.request.meta['redirect_urls'] 中找到您需要的内容.

You can find what you need in response.request.meta['redirect_urls'].

引用自 docs:

请求经过的网址(在被重定向时)可以可以在 redirect_urls Request.meta 键中找到.

The urls which the request goes through (while being redirected) can be found in the redirect_urls Request.meta key.

希望有所帮助.

这篇关于如何在scrapy中获取原始start_url(重定向前)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！