无法在 Scrapy 中定义自定义下载器中间件 [英] Unable to define custom downloader middleware in Scrapy

问题描述

我正在尝试在 Scrapy 中设置自定义下载器中间件类.我怀疑我错过了一些明显的东西，但我已经阅读了几次文档并且没有找到解决方案.我对应该是一个非常简单的任务感到有点沮丧，所以希望有人能够为我提供一些见解.

I am attempting to set up a custom downloader middleware class in Scrapy. I suspect that I've missed something obvious, but I've read over the docs a few times and have found no solutions. I'm getting a bit frustrated with what should be an extremely simple task, so hopefully someone will be able to provide me with some insight.

我已将以下行添加到我的 settings.py 文件中.

I've added the following line to my settings.py file.

DOWNLOADER_MIDDLEWARES = { 'myproject.middlewares.TestDownloader': 400 }

在添加该行并运行项目后，我收到关于模块中间件不存在的错误.经过一番研究，我发现您需要在 middlewares 文件夹中添加一个 __init__.py 文件，以便 Python 识别它.我这样做了，现在出现以下错误:

After adding that line and running the project, I got an error about the module middlewares not existing. After some research, I discovered that you needed to add a __init__.py file to the middlewares folder for Python to recognize it. I did this, and am now getting the following error:

NameError: Module 'myproject.middlewares' doesn't define any object named 'TestDownloader'

TestDownloader.py 文件没有被编译，而项目中的所有其他 *.py 文件正在编译.如果我正确理解我的 Python 阅读，这意味着它没有被导入到任何地方，但我找不到任何额外的 Scrapy 设置来更改以使其工作.

The TestDownloader.py file is not being compiled, while all other *.py files in the project are. If I understand my Python reading correctly, that means it isn't being imported anywhere, but I can't find any additional Scrapy settings to change to make this work.

推荐答案

DOWNLOADER_MIDDLEWARES = { 'myproject.middlewares.TestDownloader': 400 }

为此，在 myproject 文件夹中创建文件 middlewares.py，并在该文件中放置名为 TestDownloader 的下载器中间件类.

For this to work, create file middlewares.py inside myproject folder, and in that file put your downloader middleware class called TestDownloader.

或者在 __init__.py 里面有 middlewares 文件夹，你可以把名为 TestDownloader 的下载器中间件类放在 __init__ 里面.py -- 这也应该有效.

Or having middlewares folder with __init__.py inside it, you can put put your downloader middleware class called TestDownloader inside __init__.py -- this should work too.

这篇关于无法在 Scrapy 中定义自定义下载器中间件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！