使用 python-Scrapy 抓取动态内容 [英] Scraping dynamic content using python-Scrapy

问题描述

免责声明:我在 StackOverflow 上看到了许多其他类似的帖子，并试图以同样的方式进行，但它们似乎在本网站上不起作用.

Disclaimer: I've seen numerous other similar posts on StackOverflow and tried to do it the same way but was they don't seem to work on this website.

我正在使用 Python-Scrapy 从 koovs.com 获取数据.

I'm using Python-Scrapy for getting data from koovs.com.

但是，我无法获得动态生成的产品尺寸.具体来说，如果有人可以指导我从 这个链接，我将不胜感激.

However, I'm not able to get the product size, which is dynamically generated. Specifically, if someone could guide me a little on getting the 'Not available' size tag from the drop-down menu on this link, I'd be grateful.

我可以静态获取尺码列表，但这样做我只能获取尺码列表，而不能获取哪些尺码可用.

I am able to get the size list statically, but doing that I only get the list of sizes but not which of them are available.

推荐答案

你也可以用ScrapyJS解决(不需要 selenium 和真正的浏览器):

You can also solve it with ScrapyJS (no need for selenium and a real browser):

这个库提供了使用 Splash 的 Scrapy+JavaScript 集成.

This library provides Scrapy+JavaScript integration using Splash.

按照 Splash 和ScrapyJS，启动splash docker容器:

Follow the installation instructions for Splash and ScrapyJS, start the splash docker container:

$ docker run -p 8050:8050 scrapinghub/splash

将以下设置放入settings.py:

SPLASH_URL = 'http://192.168.59.103:8050' 

DOWNLOADER_MIDDLEWARES = {
    'scrapyjs.SplashMiddleware': 725,
}

DUPEFILTER_CLASS = 'scrapyjs.SplashAwareDupeFilter'

这是您的示例蜘蛛，它能够查看尺寸可用性信息:

And here is your sample spider that is able to see the size availability information:

# -*- coding: utf-8 -*-
import scrapy


class ExampleSpider(scrapy.Spider):
    name = "example"
    allowed_domains = ["koovs.com"]
    start_urls = (
        'http://www.koovs.com/only-onlall-stripe-ls-shirt-59554.html?from=category-651&skuid=236376',
    )

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, self.parse, meta={
                'splash': {
                    'endpoint': 'render.html',
                    'args': {'wait': 0.5}
                }
            })

    def parse(self, response):
        for option in response.css("div.select-size select.sizeOptions option")[1:]:
            print option.xpath("text()").extract()

这是控制台上打印的内容:

Here is what is printed on the console:

[u'S / 34 -- Not Available']
[u'L / 40 -- Not Available']
[u'L / 42']

这篇关于使用 python-Scrapy 抓取动态内容的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！