R 使用下拉菜单抓取 [英] R scraping with a dropdown menu

问题描述

我试图从网站上抓取 NBA 每日 ROS 预测:

I am attempting to scrape the NBA daily ROS projections from the site:https://hashtagbasketball.com/fantasy-basketball-projections.

Problem is the default number of players selected is 200, I would want 400 (or ALL would work too).

This code retrieves the first 200 no problem:

> url <- 'https://hashtagbasketball.com/fantasy-basketball-projections'
> 
> page <- read_html(url)
> 
> projs <- html_table(page)[[3]] %>% ### anything after this just cleans the df
+     rename_all(~gsub('3pm','threes',gsub('\\%','pct',tolower(.)))) %>% 
+     mutate_at(vars(matches('pct$')),~stringr::str_sub(.,1,4)) %>% 
+     mutate(player = stringr::word(player,1, 2, sep = ' ')) %>% 
+     mutate(pos = stringr::word(pos,1,1,sep = ',')) %>% 
+     mutate(pos2 = gsub('P','',pos)) %>% 
+     drop_na(player) %>% 
+     mutate_at(vars(-c(player,matches('pos'),team)),~as.numeric(.)) %>% 
+     select(player, matches('pos'),everything(),-`r#`) %>% 
+     head(2)
> projs
         player pos pos2 team gp  mpg fgpct ftpct threes  pts treb ast stl blk  to total
1  James Harden  PG    G  HOU 64 36.3  0.44  0.86    4.7 34.4  6.6 9.3 1.7 0.8 4.6 17.68
2 Anthony Davis  PF    F  LAL 65 34.8  0.50  0.84    1.3 26.6  9.4 3.2 1.5 2.3 2.5 14.56

That creates a table with all desired categories. However, when I use the code below does not extract all the statistical categories (only gp and mpg):

> pgsession <- html_session(url)
> pgform <-html_form(pgsession)[[1]]
> filled_form <-set_values(pgform,
+                          "ctl00$ContentPlaceHolder1$DDSHOW" = "400")
> 
> d <- submit_form(session=pgsession, form=filled_form)
Submitting with '<unnamed>'
> 
> y <- d %>%
+     html_nodes("table") %>%
+     .[[3]] %>%
+     html_table(header=TRUE) %>% 
+     mutate(PLAYER = stringr::word(PLAYER,1, 2, sep = ' ')) %>% 
+     head(2)
> y
  R#        PLAYER   POS TEAM GP  MPG TOTAL
1  1  James Harden PG,SG  HOU 64 36.3  0.00
2  2 Anthony Davis  PF,C  LAL 65 34.8  0.00

Any idea what I'm doing wrong? Thanks

解决方案

The problem seems to be that the check boxes for those other variables are not checked when you submit your form. You will have to set them manually. This shows you how to get ftm and ftpct. I will leave the rest to you:

library(tidyverse)
library(rvest)
url <- 'https://hashtagbasketball.com/fantasy-basketball-projections'
pgsession <- html_session(url)
pgform <-html_form(pgsession)
pgform[[1]][[5]][["ctl00$ContentPlaceHolder1$CBFTM"]]$value <- "checked" 
pgform[[1]][[5]][["ctl00$ContentPlaceHolder1$CBFTP"]]$value <- "checked" 

filled_form <-set_values(pgform[[1]],"ctl00$ContentPlaceHolder1$DDSHOW" = "400")
d <- submit_form(session=pgsession, form=filled_form)

d %>%
       html_nodes("table") %>%
       .[[3]] %>% 
       html_table() %>%
       rename_all(~gsub('3pm','threes',gsub('\\%','pct',tolower(.)))) %>% 
       mutate_at(vars(matches('pct$')),~stringr::str_sub(.,1,4)) %>% 
       mutate(player = stringr::word(player,1, 2, sep = ' ')) %>% 
       mutate(pos = stringr::word(pos,1,1,sep = ',')) %>% 
       mutate(pos2 = gsub('P','',pos)) %>% 
       drop_na(player) %>% 
       mutate_at(vars(-c(player,matches('pos'),team)),~as.numeric(.)) %>% 
       select(player, matches('pos'),everything(),-`r#`) %>% 
       head(2)
#        player pos pos2 team gp  mpg  ftm ftpct total
#1 James Harden  PG    G  HOU 64 36.3 10.4  0.86 10.95
#2 Devin Booker  SG   SG  PHX 70 35.6  6.7  0.91  7.99

In case you weren't aware, you can get the checkbox names by right clicking and choosing "Inspect" in Chrome:

这篇关于R 使用下拉菜单抓取的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！