如何重试在scrapy中找不到的404链接? [英] How to retry for 404 link not found in scrapy?

问题描述

有些网站会暂时出现 404 错误.但我粘贴在浏览器上它有效.如何告诉scrapy重试404状态码链接5次.

Some sites give 404 error temporarily. But I paste in on browser it works. How to tell scrapy to retry the 404 status code links for 5 times.

推荐答案

有两个 Scrapy 设置与您的需求相关:

There are two Scrapy settings relevant to what you need:

• RETRY_HTTP_CODES:你应该覆盖项目中的默认值以包含 404
• RETRY_TIMES:只需将其设置为5

• RETRY_HTTP_CODES: you should override the default value in your project to include 404
• RETRY_TIMES: just set it to 5

这篇关于如何重试在scrapy中找不到的404链接?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！