避免重复的 URL 抓取 [英] Avoid Duplicate URL Crawling

问题描述

我编写了一个简单的爬虫程序.在settings.py文件中，参考scrapy文档，我使用了

I coded a simple crawler. In the settings.py file, by referring to scrapy documentation, I used

DUPEFILTER_CLASS = 'scrapy.dupefilter.RFPDupeFilter'

如果我停止爬虫并再次重新启动爬虫，它会再次抓取重复的 url.我做错了什么吗?

If I stop the crawler and restart the crawler again, it is scraping the duplicate urls again. Am I doing something wrong?

推荐答案

我相信您正在寻找的是持久性支持"，即暂停和恢复爬行.

I believe what you are looking for is "persistence support", to pause and resume crawls.

要启用它，您可以:

scrapy crawl somespider -s JOBDIR=crawls/somespider-1

您可以在此处阅读更多相关信息.

You can read more about it here.

这篇关于避免重复的 URL 抓取的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！