scrapy-splash 活动内容选择器适用于 shell 但不适用于蜘蛛 [英] scrapy-splash active content selector works in shell but not with spider
本文介绍了scrapy-splash 活动内容选择器适用于 shell 但不适用于蜘蛛的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我刚开始使用 scrapy-splash 从 opentable.com 检索预订数量.以下在 shell 中工作正常:
I just started using scrapy-splash to retrieve the number of bookings from opentable.com. The following works fine in the shell:
$ scrapy shell 'http://localhost:8050/render.html?url=https://www.opentable.com/new-york-restaurant-listings&timeout=10&wait=0.5'
...
In [1]: response.css('div.booking::text').extract()
Out[1]:
['Booked 59 times today',
'Booked 20 times today',
'Booked 17 times today',
'Booked 29 times today',
'Booked 29 times today',
...
]
然而,这个简单的蜘蛛返回一个空列表:
However, this simple spider returns an empty list:
class TableSpider(scrapy.Spider):
name = 'opentable'
start_urls = ['https://www.opentable.com/new-york-restaurant-listings']
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url=url,
callback=self.parse,
endpoint='render.html',
args={'wait': 1.5},
)
def parse(self, response):
yield {'bookings': response.css('div.booking::text').extract()}
调用时:
$ scrapy crawl opentable
...
DEBUG: Scraped from <200 https://www.opentable.com/new-york-restaurant-listings>
{'bookings': []}
我已经尝试不成功
docker run -it -p 8050:8050 scrapinghub/splash --disable-private-mode
并增加等待时间.
推荐答案
我觉得你的问题出在middlewares
,首先你需要添加一些设置
I think your problem is in middlewares
, first of all you need to add some settings
# settings.py
# uncomment `DOWNLOADER_MIDDLEWARES` and add this settings to it
DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
# url of splash server
SPLASH_URL = 'http://localhost:8050'
# and some splash variables
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'
现在运行 docker
And now run docker
sudo docker run -it -p 8050:8050 scrapinghub/splash --disable-private-mode
如果我完成所有这些步骤,就会回来:
If i do all these steps a get back:
scrapy crawl opentable
...
2018-06-23 11:23:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.opentable.com/new-york-restaurant-listings via http://localhost:8050/render.html> (referer: None)
2018-06-23 11:23:54 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.opentable.com/new-york-restaurant-listings>
{'bookings': [
'Booked 44 times today',
'Booked 24 times today',
'and many others Booked values'
]}
这篇关于scrapy-splash 活动内容选择器适用于 shell 但不适用于蜘蛛的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文