无法通过scrapy访问xpath属性 [英] Having trouble accessing xpath attribute with scrapy

问题描述

我目前正在尝试抓取以下网址: http://www.bedbathandbeyond.com /store/product/dyson-dc59-motorhead-cordless-vacuum/1042997979?categoryId = 10562

I am currently trying to scrape the following url: http://www.bedbathandbeyond.com/store/product/dyson-dc59-motorhead-cordless-vacuum/1042997979?categoryId=10562

在此页面上，我要提取列出的评论数.也就是说，我要提取数字693.

On this page, I want to extract the number of reviews listed. That is, I want to extract the number 693.

这是我当前的xpath:

This is my current xpath:

sel.xpath('//*[@id="BVRRRatingSummaryLinkReadID"]/a/span/span')

似乎只返回一个空数组，有人可以建议正确的xpath吗?

It seems to be only returning an empty array, can someone suggest a correct xpath?

推荐答案

使用Scrapy获得的初始页面上没有任何评论.问题在于，评论是通过大量使用javascript来加载和构建的，这使事情变得更加复杂.

There are no reviews on the initial page you are getting with Scrapy. The problem is that the reviews are loaded and constructed via the heavy use of javascript which makes things more complicated.

基本上，您的选择是:

• 一种高级方法(例如，使用具有 selenium 的真实浏览器) .您甚至可以结合使用Scrapy和Selenium:
  • 含硒的硒对动态页面的影响
  • 使用Scrapy和Selenium进行刮擦
  • scrapy-webdriver
  • a high-level approach (for example, use a real browser with selenium). You can even combine Scrapy and Selenium:
    • selenium with scrapy for dynamic page
    • Scraping with Scrapy and Selenium
    • scrapy-webdriver

    这是涉及使用 json 和 slimit ，从中提取HTML并进行解析通过 BeautifulSoup :

    Here is a working example of the low-level approach involving parsing of a javascript code with json and slimit, extracting HTML from it and parsing it via BeautifulSoup:

    import json
    
    from bs4 import BeautifulSoup
    import requests
    from slimit import ast
    from slimit.parser import Parser
    from slimit.visitors import nodevisitor
    
    ID = 1042997979
    
    url = 'http://bedbathandbeyond.ugc.bazaarvoice.com/2009-en_us/{id}/reviews.djs?format=embeddedhtml&sort=submissionTime'.format(id=ID)
    
    response = requests.get(url)
    
    parser = Parser()
    tree = parser.parse(response.content)
    data = ""
    for node in nodevisitor.visit(tree):
        if isinstance(node, ast.Object):
            data = json.loads(node.to_ecma())
            if "BVRRSourceID" in data:
                break
    
    soup = BeautifulSoup(data['BVRRSourceID'])
    print soup.select('span.BVRRCount span.BVRRNumber')[0].text
    

    打印693.

    要使解决方案适应Scrapy，您需要使用Scrapy而不是requests进行请求，并使用Scrapy而不是BeautifulSoup来解析HTML.

    To adapt the solution to Scrapy, you would need to make a request with Scrapy instead of requests, and parse the HTML with Scrapy instead of BeautifulSoup.

    这篇关于无法通过scrapy访问xpath属性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！