使用“加载更多"发出抓取页面按钮 [英] Issue scraping page with "Load more" button with rvest

问题描述

我想获取此页面上列出的自动取款机的链接:https://coinatmradar.com/city/345/bitcoin-atm-birmingham-uk/

I want to obtain the links to the atms listed on this page: https://coinatmradar.com/city/345/bitcoin-atm-birmingham-uk/

我需要对页面底部的加载更多"按钮做些什么吗?

Would I need to do something about the 'load more' button at the bottom of the page?

我一直在使用您可以为 chrome 下载的选择器工具来选择 CSS 路径.

I have been using the selector tool you can download for chrome to select the CSS path.

我写了下面的代码块，它似乎只检索前十个链接.

I've written the below code block and it only seems to retrieve the first ten links.

library(rvest)

base <- "https://coinatmradar.com/city/345/bitcoin-atm-birmingham-uk/"
base_read <- read_html(base)
atm_urls <- html_nodes(base_read, ".place > a")
all_urls_final <- html_attr(atm_urls, "href" )
print(all_urls_final)

我希望能够检索到该区域中列出的自动取款机的所有链接，但我的 R 代码没有这样做.

I expected to be able to retrieve all links to the atms listed in the area but my R code has not done so.

任何帮助都会很棒.对不起，如果这是一个非常简单的问题.

Any help would be great. Sorry if this is a really simple question.

推荐答案

你应该给 RSelenium 一试.我可以使用以下代码获取链接:

You should give RSelenium a try. I'm able to get the links with the following code:

# install.packages("RSelenium")
library(RSelenium)
library(rvest)

# Download binaries, start driver, and get client object.
rd <- rsDriver(browser = "firefox", port = 4444L)
ffd <- rd$client

# Navigate to page.
ffd$navigate("https://coinatmradar.com/city/345/bitcoin-atm-birmingham-uk/")

# Find the load button and assign, then send click event.
load_btn <- ffd$findElement(using = "css selector", ".load-more .btn")
load_btn$clickElement()

# Wait for elements to load.
Sys.sleep(2)

# Get HTML data and parse
html_data <- ffd$getPageSource()[[1]]
html_data %>% 
    read_html() %>% 
    html_nodes(".place a:not(.operator-link)") %>% 
    html_attr("href")

#### OUTPUT ####

#  [1] "/bitcoin_atm/5969/bitcoin-atm-shitcoins-club-birmingham-uk-bitcoin-embassy/"                   
#  [2] "/bitcoin_atm/7105/bitcoin-atm-general-bytes-northampton-costcutter/"                           
#  [3] "/bitcoin_atm/4759/bitcoin-atm-general-bytes-birmingham-uk-costcutter/"                         
#  [4] "/bitcoin_atm/2533/bitcoin-atm-general-bytes-birmingham-uk-londis-# convenience/"                 
#  [5] "/bitcoin_atm/5458/bitcoin-atm-general-bytes-coventry-agg-african-restaurant/"                  
#  [6] "/bitcoin_atm/711/bitcoin-atm-general-bytes-coventry-bigs-barbers/"                             
#  [7] "/bitcoin_atm/5830/bitcoin-atm-general-bytes-telford-bpred-lion-service-station/"               
#  [8] "/bitcoin_atm/5466/bitcoin-atm-general-bytes-nottingham-24-express-off-licence/"                
#  [9] "/bitcoin_atm/4615/bitcoin-atm-general-bytes-northampton-costcutter/"                           
# [10] "/bitcoin_atm/4841/bitcoin-atm-lamassu-worcester-computer-house/"                               
# [11] "/bitcoin_atm/3150/bitcoin-atm-bitxatm-leicester-keshs-wines-and-newsagents-braustone/"         
# [12] "/bitcoin_atm/2948/bitcoin-atm-bitxatm-coventry-nisa-local/"                                    
# [13] "/bitcoin_atm/4742/bitcoin-atm-bitxatm-birmingham-uk-custcutter-coventry-road-hay-mills/"       
# [14] "/bitcoin_atm/4741/bitcoin-atm-bitxatm-derby-michaels-drink-store-alvaston/"                    
# [15] "/bitcoin_atm/4740/bitcoin-atm-bitxatm-birmingham-uk-nisa-local-crabtree-# hockley/"              
# [16] "/bitcoin_atm/4739/bitcoin-atm-bitxatm-birmingham-uk-nisa-local-subway-boldmere/"               
# [17] "/bitcoin_atm/4738/bitcoin-atm-bitxatm-birmingham-uk-ashtree-convenience-store/"                
# [18] "/bitcoin_atm/4737/bitcoin-atm-bitxatm-birmingham-uk-nisa-local-finnemore-road-bordesley-green/"
# [19] "/bitcoin_atm/3160/bitcoin-atm-bitxatm-birmingham-uk-costcutter/" 

这篇关于使用“加载更多"发出抓取页面按钮的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！