在scrapy中排队后请求消失 [英] requests disappear after queueing in scrapy

问题描述

Scrapy 似乎没有处理所有请求就完成了.我知道这一点是因为我在请求排队之前和之后都进行了日志记录，我可以清楚地看到这一点.

Scrapy seems to complete without processing all the requests. I know this because i am logging before and after queueing the request and I can clearly see that.

我同时登录了解析和错误回调方法，但没有一个被调用来处理那些丢失的请求.

I am logging in both parse and error callback methods and none of them got called for those missing requests.

如何调试这些请求发生了什么?

How can I debug what happened to those requests?

推荐答案

重新排队请求时需要添加dont_filter=True.尽管该请求可能与其他请求不匹配，但 Scrapy 会记住它已经发出的请求，如果您重新排队，它会过滤掉.它会假设这是错误的.

You need to add dont_filter=True when re-queueing the request. Though the request may not match other request but Scrapy remembers what requests it has already made and it will filter out if you re-queue it. It will assume it was by mistake.

这篇关于在scrapy中排队后请求消失的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！