如何在爬行中动态生成start_urls? [英] How to generate the start_urls dynamically in crawling?

问题描述

我正在抓取一个可能包含很多 start_urls 的网站，例如:

http://www.a.com/list_1_2_3.htm

我想像 [list_\d+_\d+_\d+\.htm] 一样填充 start_urls，并在抓取过程中从 [node_\d+\.htm] 等 URL 中提取项目.

我可以使用CrawlSpider来实现这个功能吗?以及如何在爬行中动态生成 start_urls?

解决方案

动态生成 URL 的最佳方式是覆盖 start_requests 蜘蛛的方法:

<块引用>

from scrapy.http.request 导入请求def start_requests(self):使用 open('urls.txt', 'rb') 作为网址:对于网址中的网址:产量请求(网址，self.parse)

I am crawling a site which may contain a lot of start_urls, like:

http://www.a.com/list_1_2_3.htm

I want to populate start_urls like [list_\d+_\d+_\d+\.htm], and extract items from URLs like [node_\d+\.htm] during crawling.

Can I use CrawlSpider to realize this function? And how can I generate the start_urls dynamically in crawling?

解决方案

The best way to generate URLs dynamically is to override the start_requests method of the spider:

from scrapy.http.request import Request

def start_requests(self):
      with open('urls.txt', 'rb') as urls:
          for url in urls:
              yield Request(url, self.parse)

这篇关于如何在爬行中动态生成start_urls?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！