python中的Scrapy Crawler无法跟踪链接? [英] Scrapy Crawler in python cannot follow links?
问题描述
我使用python的scrapy工具用python写了一个爬虫.以下是python代码:
I wrote a crawler in python using the scrapy tool of python. The following is the python code:
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
#from scrapy.item import Item
from a11ypi.items import AYpiItem
class AYpiSpider(CrawlSpider):
name = "AYpi"
allowed_domains = ["a11y.in"]
start_urls = ["http://a11y.in/a11ypi/idea/firesafety.html"]
rules =(
Rule(SgmlLinkExtractor(allow = ()) ,callback = 'parse_item')
)
def parse_item(self,response):
#filename = response.url.split("/")[-1]
#open(filename,'wb').write(response.body)
#testing codes ^ (the above)
hxs = HtmlXPathSelector(response)
item = AYpiItem()
item["foruri"] = hxs.select("//@foruri").extract()
item["thisurl"] = response.url
item["thisid"] = hxs.select("//@foruri/../@id").extract()
item["rec"] = hxs.select("//@foruri/../@rec").extract()
return item
但是,抛出的错误不是跟随链接:
But, instead of following the links the error thrown is:
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/Scrapy-0.12.0.2538-py2.6.egg/scrapy/cmdline.py", line 131, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "/usr/lib/python2.6/site-packages/Scrapy-0.12.0.2538-py2.6.egg/scrapy/cmdline.py", line 97, in _run_print_help
func(*a, **kw)
File "/usr/lib/python2.6/site-packages/Scrapy-0.12.0.2538-py2.6.egg/scrapy/cmdline.py", line 138, in _run_command
cmd.run(args, opts)
File "/usr/lib/python2.6/site-packages/Scrapy-0.12.0.2538-py2.6.egg/scrapy/commands/crawl.py", line 45, in run
q.append_spider_name(name, **opts.spargs)
--- <exception caught here> ---
File "/usr/lib/python2.6/site-packages/Scrapy-0.12.0.2538-py2.6.egg/scrapy/queue.py", line 89, in append_spider_name
spider = self._spiders.create(name, **spider_kwargs)
File "/usr/lib/python2.6/site-packages/Scrapy-0.12.0.2538-py2.6.egg/scrapy/spidermanager.py", line 36, in create
return self._spiders[spider_name](**spider_kwargs)
File "/usr/lib/python2.6/site-packages/Scrapy-0.12.0.2538-py2.6.egg/scrapy/contrib/spiders/crawl.py", line 38, in __init__
self._compile_rules()
File "/usr/lib/python2.6/site-packages/Scrapy-0.12.0.2538-py2.6.egg/scrapy/contrib/spiders/crawl.py", line 82, in _compile_rules
self._rules = [copy.copy(r) for r in self.rules]
exceptions.TypeError: 'Rule' object is not iterable
有人可以向我解释发生了什么吗?由于这是文档中提到的内容并且我将允许字段留空,因此默认情况下它本身应该遵循 True .那么为什么会出错呢?我可以对我的爬虫进行哪些优化以使其更快?
Can someone please explain to me what's going on? Since this is the stuff mentioned in the documentation and I leave the allow field blank, that itself should make follow True by default. So why the error? What kind of optimisations can I make with my crawler to make it fast?
推荐答案
从我看来,您的规则似乎不是可迭代的.看起来您试图将规则设为元组,您应该 阅读了解 Python 文档中的元组.
From what I see, it looks like your rule is not an iterable. It looks like you were trying to make rules a tuple, you should read up on tuples in the python documentation.
要解决您的问题,请更改此行:
To fix your problem, change this line:
rules =(
Rule(SgmlLinkExtractor(allow = ()) ,callback = 'parse_item')
)
致:
rules =(Rule(SgmlLinkExtractor(allow = ()) ,callback = 'parse_item'),)
注意到末尾的逗号了吗?
Notice the comma at the end?
这篇关于python中的Scrapy Crawler无法跟踪链接?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!