CrawlSpider 与飞溅 [英] CrawlSpider with Splash
问题描述
我的蜘蛛有问题.我使用带有scrapy的飞溅来获取到由JavaScript生成的下一页"的链接.从第一页下载信息后,我想从以下页面下载信息,但LinkExtractor功能无法正常工作.但看起来 start_request 函数不起作用.这是代码:
I have some problem with my spider. I use splash with scrapy to get link to "Next page" which is generate by JavaScript. After downloading the information from the first page, I want to download information from the following pages, but LinkExtractor function does not work properly. But it looks like start_request function doesn't work. Here is code:
class ReutersBusinessSpider(CrawlSpider):
name = 'reuters_business'
allowed_domains = ["reuters.com"]
start_urls = (
'http://reuters.com/news/archive/businessNews?view=page&page=1',
)
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(url, self.parse, meta={
'splash': {
'endpoint': 'render.html',
'args': {'wait': 0.5}
}
})
def use_splash(self, request):
request.meta['splash'] = {
'endpoint':'render.html',
'args':{
'wait':0.5,
}
}
return request
def process_value(value):
m = re.search(r'(\?view=page&page=[0-9]&pageSize=10)', value)
if m:
return urlparse.urljoin('http://reuters.com/news/archive/businessNews',m.group(1))
rules = (
Rule(LinkExtractor(restrict_xpaths='//*[@class="pageNext"]',process_value='process_value'),process_request='use_splash', follow=False),
Rule(LinkExtractor(restrict_xpaths='//h2/*[contains(@href,"article")]',process_value='process_value'),callback='parse_item'),
)
def parse_item(self, response):
l = ItemLoader(item=PajaczekItem(), response=response)
l.add_xpath('articlesection','//span[@class="article-section"]/text()', MapCompose(unicode.strip), Join())
l.add_xpath('date','//span[@class="timestamp"]/text()', MapCompose(parse))
l.add_value('url',response.url)
l.add_xpath('articleheadline','//h1[@class="article-headline"]/text()', MapCompose(unicode.title))
l.add_xpath('articlelocation','//span[@class="location"]/text()')
l.add_xpath('articletext','//span[@id="articleText"]//p//text()', MapCompose(unicode.strip), Join())
return l.load_item()
日志:
2016-02-12 08:20:29 [scrapy] INFO: Spider opened 2016-02-12 08:20:29 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-02-12 08:20:29 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-02-12 08:20:38 [scrapy] DEBUG: Crawled (200) <POST localhost:8050/render.html>; (referer: None)
2016-02-12 08:20:38 [scrapy] DEBUG: Filtered offsite request to 'localhost': <GET http://localhost:8050/render.html?page=2&pageSize=10&view=page%3E;
2016-02-12 08:20:38 [scrapy] INFO: Closing spider (finished)
错误在哪里?感谢您的帮助.
Where is mistake? Thanks for help.
推荐答案
快速浏览,您并没有使用 splash 调用 start_request 属性...例如,您应该使用 SplashRequest.
A quick glance, you're not calling your start_request property using splash... For example, you should be using SplashRequest.
def start_requests(self):
for url in self.start_urls:
yield SplahRequest(url, self.parse, meta={
'splash': {
'endpoint': 'render.html',
'args': {'wait': 0.5}
}
})
假设您已经适当地设置了 Splash,即在设置中您启用了必要的中间位置并指向正确的/url 也使它们能够正确触发和 HTTP 缓存...不,我还没有运行您的代码现在应该可以走了
Giving that you have Splash set up appropriate, that is in settings you have enabled the necessary middle where's and pointed to the correct /url also enabled them to fire and HTTP cache all correctly... No I have not run your code should be good to go now
所以...除非您使用飞溅有任何其他原因,否则我认为没有理由在文章请求的初始解析中使用它作为简单的 for 循环,例如...
So... unless there is any other reason your using splash I see no reason to use it a simple for loop in the initial parsing of the articles request like...
for next in response.css("a.control-nav-next::attr(href)").extract():
yield scrapy.Request(response.urljoin(next), callback=self.parse...
这篇关于CrawlSpider 与飞溅的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!