NameError:名称“规则"未在python scrapy中定义 [英] NameError: name 'Rule' is not defined in python scrapy

问题描述

我有以下用于递归抓取网站的脚本:

I have the following script for crawling a website recursively:

#!/usr/bin/python 
import scrapy
from scrapy.selector import Selector
from twisted.internet import reactor
from scrapy.crawler import CrawlerRunner

class GivenSpider(scrapy.Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/",
#        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
 #       "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]
    rules = (Rule(LinkExtractor(allow=r'/'), callback=parse, follow=True),)

    def parse(self, response):
        select = Selector(response)
        titles = select.xpath('//a[@class="listinglink"]/text()').extract()
        print ' [*] Start crawling at %s ' % response.url
        for title in titles:
            print '\t %s' % title


#configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})
runner = CrawlerRunner()

d = runner.crawl(GivenSpider)
d.addBoth(lambda _: reactor.stop())
reactor.run()

当我调用它时:

$ python spide.py
NameError: name 'Rule' is not defined

推荐答案

如果您查看文档并搜索规则"一词，您会发现:

If you go by the documentation and search for the word Rule, you'll find this:

http://doc.scrapy.org/en/0.20/topics/spiders.html?highlight=rule#crawling-rules

由于您没有导入任何内容，很明显没有定义 Rule.

As you didn't import anything, it is clear that Rule isn't being defined.

 class scrapy.contrib.spiders.Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, process_links=None, process_request=None)

因此，理论上，您应该能够使用 from scrapy.contrib.spiders import Rule

So, in theory, you should be able to import the Rule class with from scrapy.contrib.spiders import Rule

这篇关于NameError:名称“规则"未在python scrapy中定义的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！