rvest 网络抓取在尝试收集产品价格信息时返回一个空数据框 [英] rvest web scraping is returning an empty data frame when attempting to collect product price information

问题描述

我正在尝试使用rvest"从以下位置获取产品定价:

I am trying to use 'rvest' to scrape product pricing from: https://www.lowes.com/pl/Lawn-garden-hand-tools-Outdoor-tools-equipment-Outdoors/4294612737?goToProdList=true&int_cmp=LawnGardenHandTools:C:Outdoors:Merch:shop_all_copy. I am using the below code:

library(rvest)
library(tidyverse)

url <- "https://www.lowes.com/pl/Lawn-garden-hand-tools-Outdoor-tools-equipment-Outdoors/4294612737?goToProdList=true&int_cmp=LawnGardenHandTools:C:Outdoors:Merch:shop_all_copy"

html <- read_html(url)

price <- html %>%
  html_node('body') %>%
  xml_find_all("//span[contains(@class, 'h5 js-price v-spacing-mini art-pl-price')]") %>% 
  html_text() %>%
  data.frame()

However - this is returning an empty data frame.

Any advice would be much appreciated.

解决方案

rvest can only scrape static HTML content.

Most modern commercial websites use dynamic web content generated on the fly by a JavaScript.

In order to scrape such websites, you will first need to make the site generate the HTML content you are looking for, and then you'll be able to scrape it with rvest.

To do so, you'll need to use a web browser emulator like RSelenium or Splash to allow you to move around in the site and query data programmatically.

RSelenium needs installation of a Docker Selenium server. This is also recommended for Splash.

At the end of this long and interesting journey, you'll have to be creative so that the Website doesn't think that it's being queried by a robot :

这篇关于rvest 网络抓取在尝试收集产品价格信息时返回一个空数据框的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！