从脚本运行 Scrapy - 挂起 [英] Running Scrapy from a script - Hangs
本文介绍了从脚本运行 Scrapy - 挂起的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我正在尝试从此处讨论的脚本运行scrapy.它建议使用 this 片段,但是当我这样做时它会无限期地挂起.这是在 .10 版中写回的;是否仍与当前稳定版兼容?
I'm trying to run scrapy from a script as discussed here. It suggested using this snippet, but when I do it hangs indefinitely. This was written back in version .10; is it still compatible with the current stable?
推荐答案
from scrapy import signals, log
from scrapy.xlib.pydispatch import dispatcher
from scrapy.crawler import CrawlerProcess
from scrapy.conf import settings
from scrapy.http import Request
def handleSpiderIdle(spider):
'''Handle spider idle event.''' # http://doc.scrapy.org/topics/signals.html#spider-idle
print '\nSpider idle: %s. Restarting it... ' % spider.name
for url in spider.start_urls: # reschedule start urls
spider.crawler.engine.crawl(Request(url, dont_filter=True), spider)
mySettings = {'LOG_ENABLED': True, 'ITEM_PIPELINES': 'mybot.pipeline.validate.ValidateMyItem'} # global settings http://doc.scrapy.org/topics/settings.html
settings.overrides.update(mySettings)
crawlerProcess = CrawlerProcess(settings)
crawlerProcess.install()
crawlerProcess.configure()
class MySpider(BaseSpider):
start_urls = ['http://site_to_scrape']
def parse(self, response):
yield item
spider = MySpider() # create a spider ourselves
crawlerProcess.queue.append_spider(spider) # add it to spiders pool
dispatcher.connect(handleSpiderIdle, signals.spider_idle) # use this if you need to handle idle event (restart spider?)
log.start() # depends on LOG_ENABLED
print "Starting crawler."
crawlerProcess.start()
print "Crawler stopped."
更新:
如果您还需要对每个蜘蛛进行设置,请参阅此示例:
If you need to have also settings per spider see this example:
for spiderConfig in spiderConfigs:
spiderConfig = spiderConfig.copy() # a dictionary similar to the one with global settings above
spiderName = spiderConfig.pop('name') # name of the spider is in the configs - i can use the same spider in several instances - giving them different names
spiderModuleName = spiderConfig.pop('spiderClass') # module with the spider is in the settings
spiderModule = __import__(spiderModuleName, {}, {}, ['']) # import that module
SpiderClass = spiderModule.Spider # spider class is named 'Spider'
spider = SpiderClass(name = spiderName, **spiderConfig) # create the spider with given particular settings
crawlerProcess.queue.append_spider(spider) # add the spider to spider pool
文件中的蜘蛛设置示例:
Example of settings in the file for spiders:
name = punderhere_com
allowed_domains = plunderhere.com
spiderClass = scraper.spiders.plunderhere_com
start_urls = http://www.plunderhere.com/categories.php?
这篇关于从脚本运行 Scrapy - 挂起的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文