无法运行Scrapy程序 [英] Can't run Scrapy program
问题描述
我一直在下面的链接中学习如何使用Scrapy:
I have been learning how to work with Scrapy from the following link :
http://doc.scrapy.org/en/master/intro/tutorial.html
当我尝试运行在Crawling(scrapy crawl dmoz
)节中编写的代码时,出现以下错误:
When i try to run the code written in the Crawling(scrapy crawl dmoz
) section, i get the following error:
AttributeError: 'module' object has no attribute 'Spider
'
但是,我将"Spider"更改为"spider",除了一个新错误外,我什么也没有:
However, i changed "Spider" to "spider" and i got nothing but a new error:
TypeError: Error when calling the metaclass bases
module.__init__() takes at most 2 arguments (3 given)
我很困惑,这是什么问题?任何帮助将不胜感激.谢谢.顺便说一句,我正在使用Windows.
I'm so confused, what is the problem? Any help would highly be appreciated. Thanks. By the way, i am using Windows.
编辑(添加源):
首先,我使用Scrapy创建了一个项目,方法是转到目录并像这样通过cmd运行以下命令:
First i created a project using Scrapy by going to a directory and running the following commands by cmd like so :
cd #DIRECTORY PATH#
scrapy startproject tutorial
这将在给定目录中创建一个名为tutorial的文件夹.教程文件夹包括:
This will create a folder named tutorial in the given directory. The tutorial folder consists :
教程/ scrapy.cfg 教程/ 初始化 .py items.py pipelines.py settings.py 蜘蛛/ 初始化 .py ...
tutorial/ scrapy.cfg tutorial/ init.py items.py pipelines.py settings.py spiders/ init.py ...
然后我定义了我的物品:
Then i defined my items :
import scrapy
class DmozItem(scrapy.Item):
title = scrapy.Field()
link = scrapy.Field()
desc = scrapy.Field()
然后,我创建了蜘蛛:
进口沙皮
class DmozSpider(scrapy.Spider):
name = "dmoz"
allowed_domains = ["dmoz.org"]
start_urls = [
"http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
]
def parse(self, response):
filename = response.url.split("/")[-2]
with open(filename, 'wb') as f:
f.write(response.body)
然后,在运行代码时,显示错误.我正在使用Windows 7 64位和Python 2.7 32位.
And after, when running the code,the error is shown. I am using windows 7 64 bit along with Python 2.7 32 bit.
我尝试卸载并安装另一个Scrapy版本,但没有成功.这是日志:
I tried uninstalling and installing another Scrapy version but it didn't work. Here is the log:
C:\Users\Novin Pendar\Desktop\FS\tutorial>scrapy crawl dmoz
2015-03-26 17:48:29+0430 [scrapy] INFO: Scrapy 0.16.5 started (bot: tutorial)
2015-03-26 17:48:29+0430 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetCon
sole, CloseSpider, WebService, CoreStats, SpiderState
C:\Python27\lib\site-packages\scrapy-0.16.5-py2.7.egg\scrapy\__init__.pyc
Traceback (most recent call last):
File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "C:\Python27\lib\runpy.py", line 72, in _run_code
exec code in run_globals
File "C:\Python27\lib\site-packages\scrapy-0.16.5-py2.7.egg\scrapy\cmdline.py"
, line 156, in <module>
execute()
File "C:\Python27\lib\site-packages\scrapy-0.16.5-py2.7.egg\scrapy\cmdline.py"
, line 131, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "C:\Python27\lib\site-packages\scrapy-0.16.5-py2.7.egg\scrapy\cmdline.py"
, line 76, in _run_print_help
func(*a, **kw)
File "C:\Python27\lib\site-packages\scrapy-0.16.5-py2.7.egg\scrapy\cmdline.py"
, line 138, in _run_command
cmd.run(args, opts)
File "C:\Python27\lib\site-packages\scrapy-0.16.5-py2.7.egg\scrapy\commands\cr
awl.py", line 43, in run
spider = self.crawler.spiders.create(spname, **opts.spargs)
File "C:\Python27\lib\site-packages\scrapy-0.16.5-py2.7.egg\scrapy\command.py"
, line 33, in crawler
self._crawler.configure()
File "C:\Python27\lib\site-packages\scrapy-0.16.5-py2.7.egg\scrapy\crawler.py"
, line 40, in configure
self.spiders = spman_cls.from_crawler(self)
File "C:\Python27\lib\site-packages\scrapy-0.16.5-py2.7.egg\scrapy\spidermanag
er.py", line 35, in from_crawler
sm = cls.from_settings(crawler.settings)
File "C:\Python27\lib\site-packages\scrapy-0.16.5-py2.7.egg\scrapy\spidermanag
er.py", line 31, in from_settings
return cls(settings.getlist('SPIDER_MODULES'))
File "C:\Python27\lib\site-packages\scrapy-0.16.5-py2.7.egg\scrapy\spidermanag
er.py", line 22, in __init__
for module in walk_modules(name):
File "C:\Python27\lib\site-packages\scrapy-0.16.5-py2.7.egg\scrapy\utils\misc.
py", line 65, in walk_modules
submod = __import__(fullpath, {}, {}, [''])
File "tutorial\spiders\dmoz_spider.py", line 3, in <module>
class DmozSpider(scrapy.Spider):
AttributeError: 'module' object has no attribute 'Spider'
问题已解决.我下载并安装了最新版本的Scrapy(0.24).一切都很好.只是想对那些曾经遇到过同样问题的人说,这样,他们可以节省很多时间.谢谢.
The problem is solved. I downloaded the latest version of Scrapy(0.24) and installed. Everything worked so great. Just wanted to say to people who have the same problem that i used to, so, they will save them a lot of time. Thanks.
推荐答案
如果安装正确,请尝试
检查工作文件夹中的任何scrapy.py
或scrapy.pyc
如果存在,请重命名不要将Spider
更改为spider
Check any scrapy.py
or scrapy.pyc
, in the working folder.If exists, rename it.Dont change Spider
to spider
这篇关于无法运行Scrapy程序的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!