如何在scrapy中的每个请求之间提供延迟? [英] How to give delay between each requests in scrapy?

问题描述

我不想同时爬行并被阻止.我想每秒发送一个请求.

I don't want to crawl simultaneously and get blocked. I would like to send one request per second.

推荐答案

有一个 设置:

DOWNLOAD_DELAY

默认:0

下载器应该等待的时间(以秒为单位)从同一网站下载连续的页面.这个可以用限制爬取速度，避免对服务器造成太大冲击.

The amount of time (in secs) that the downloader should wait before downloading consecutive pages from the same website. This can be used to throttle the crawling speed to avoid hitting servers too hard.

DOWNLOAD_DELAY = 0.25    # 250 ms of delay

阅读文档:https://doc.scrapy.org/en/latest/index.html

这篇关于如何在scrapy中的每个请求之间提供延迟?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！