如何从 URL 中只有月份和年份的网站中的多个表中提取内容 [英] How to extract contents from multiple tables from website with only month and year in URL
问题描述
这是我上一个问题的后续:
<块引用>为什么我会遇到这个错误:Can't recycle 'Date' (size 200) to match '..3' (size 190)?
因为有如下表格(另见这个
这与您描述的排名和统计表的行数始终相同.
This is as follow up to my previous question here:
How to extract contents between div tags with rvest and then bind rows
The page that I am trying to extract the data from between the div tags is from this site:
http://bigbashboard.com/rankings/batsmen
This is a different page to my previous question (although it is still the same site). The key difference is that the dates that appear in the URL are only displayed as year/month like so:
http://bigbashboard.com/rankings/batsmen/2020/10
as opposed to the page in my previous question which appears with year/month/day like this:
http://bigbashboard.com/rankings/bbl/batsmen/2020/01/08
I am still looking to extract the same data from the left hand side of the page which appears between div tags that looks like this:
Batsmen
1 Lokesh Rahul 167
2 Ravija Sanaruwan 150
3 David Warner 143
I also need the data that appears in the table to the right and bind them together so it looks like this, including the date that page has come from like so:
Date Rank Name Points Dates I R HS Ave SR 4s 6s 100s 50s
Oct-20 1 Lokesh Rahul 167 Nov 2018 - Oct 2020 47 1910 132 50.26 141.38 171 76 2 17
Oct-20 2 Ravija Sanaruwan 150 Jan 2019 - Feb 2020 15 577 103 44.38 165.80 52 36 1 4
Oct-20 3 David Warner 143 Jan 2019 - Sep 2020 33 1475 100 61.46 138.89 128 39 2 16
I have attempted to use the code offered in the previous post as a solution:
library(rvest)
library(xml2)
library(dplyr)
library(furrr)
batsmen <- function(x) {
x <- html_nodes(x, "div.cf.rankings-page div div ol li a")
xml_remove(html_nodes(x, "span.rank small, span[class^='pos'] em"))
score <- html_text(html_nodes(x, "span.rank"))
rank <- html_text(html_nodes(x, "span[class^='pos']"), trim = TRUE)
xml_remove(html_nodes(x, "span"))
tibble(Rank = rank, Name = html_text(x), Points = score)
}
stats_table <- function(x) {
as_tibble(html_table(x)[[1L]])
}
read_rankings <- function(url) {
ymd <- as.Date(paste0(tail(strsplit(url, "/")[[1L]], 3L), collapse = "-"))
read_html(url) %>% {bind_cols(Date = ymd, batsmen(.), stats_table(.))}
}
mas_url <- "http://bigbashboard.com/rankings/batsmen"
timeline <-
read_html(mas_url) %>%
html_nodes("div.timeline span a") %>%
html_attr("href") %>%
url_absolute(mas_url)
# Use parallel processing for speed.
plan(multiprocess)
future_map_dfr(timeline[1:100], read_rankings) # I only scrape a few links for test.
However, this yields an error:
Error in charToDate(x) :
character string is not in a standard unambiguous format
I cannot understand why this occurs and how to resolve it. I am assuming it is perhaps because the dates are in a different format.
The code below works for all three cases
library(rvest)
library(xml2)
library(dplyr)
library(furrr)
batsmen <- function(x) {
nms <- html_attr(html_nodes(x, "div.cf > a"), "name")
x <- html_nodes(x, "div.cf.rankings-page")
xml_remove(html_nodes(x, "li span.rank small, li span[class^='pos'] em"))
x <- Map(function(i, nm) {
i <- html_nodes(i, "li a")
score <- html_text(html_nodes(i, "span.rank"))
rank <- html_text(html_nodes(i, "span[class^='pos']"), trim = TRUE)
xml_remove(html_nodes(i, "span"))
tibble(Title = nm, Rank = rank, Name = html_text(i), Points = score)
}, x, nms)
bind_rows(x)
}
stats_table <- function(x) {
as_tibble(bind_rows(
lapply(html_table(x), function(df) setNames(df, make.unique(names(df))))
))
}
timeline <- function(mas_url) {
links <- read_html(mas_url) %>% html_nodes("div.timeline span a")
out <- links %>% html_attr("href") %>% url_absolute(mas_url)
setNames(out, html_text(links))
}
read_rankings <- function(url, time) {
read_html(url) %>% {bind_cols(Date = time, batsmen(.), stats_table(.))}
}
# Use parallel processing for speed.
plan(multiprocess)
Case 1: only men's ranking on that page
# men only
future_imap_dfr(timeline("http://bigbashboard.com/rankings/bbl/batsmen")[1:10], ~read_rankings(.x, .y))
Output
# A tibble: 996 x 15
Date Title Rank Name Points Dates I R HS Ave SR `4s` `6s` `100s` `50s`
<chr> <chr> <chr> <chr> <chr> <chr> <int> <int> <int> <dbl> <dbl> <int> <int> <int> <int>
1 8 Feb '20 men 1 Matthew Wade 125 22 Dec 2018 - 30 Jan 2020 23 943 130 44.9 155. 78 36 1 9
2 8 Feb '20 men 2 Marcus Stoinis 120 21 Dec 2018 - 08 Feb 2020 30 1238 147 53.8 134. 111 39 1 10
3 8 Feb '20 men 3 D'Arcy Short 116 22 Dec 2018 - 30 Jan 2020 24 994 103 49.7 137. 93 36 1 9
4 8 Feb '20 men 4 Alex Hales 115 17 Dec 2019 - 06 Feb 2020 17 576 85 38.4 147. 59 23 0 6
5 8 Feb '20 men 5 Aaron Finch 89 07 Jan 2019 - 27 Jan 2020 17 583 109 36.4 130. 41 24 1 4
6 8 Feb '20 men 6 Josh Inglis 87 26 Dec 2018 - 26 Jan 2020 18 517 73 28.7 149. 53 19 0 5
7 8 Feb '20 men 7 Travis Head 87 11 Jan 2019 - 01 Feb 2020 10 291 79 29.1 132. 22 13 0 1
8 8 Feb '20 men 8 Josh Philippe 84 22 Dec 2018 - 08 Feb 2020 31 791 86 34.4 140. 76 23 0 7
9 8 Feb '20 men 9 Shaun Marsh 82 24 Jan 2019 - 21 Jan 2020 15 547 96 39.1 128. 45 19 0 4
10 8 Feb '20 men 10 Chris Lynn 78 19 Dec 2018 - 27 Jan 2020 27 772 94 32.2 137. 64 44 0 6
# ... with 986 more rows
Case 2: men's and women's rankings on the same page
# men and women
future_imap_dfr(timeline("http://bigbashboard.com/rankings/batsmen")[1:10], ~read_rankings(.x, .y))
# A tibble: 2,000 x 15
Date Title Rank Name Points Dates I R HS Ave SR `4s` `6s` `100s` `50s`
<chr> <chr> <chr> <chr> <chr> <chr> <int> <int> <int> <dbl> <dbl> <int> <int> <int> <int>
1 Oct '20 men 1 Lokesh Rahul 167 Nov 2018 - Oct 2020 47 1910 132 50.3 141. 171 76 2 17
2 Oct '20 men 2 Ravija Sandaruwan 150 Jan 2019 - Feb 2020 15 577 103 44.4 166. 52 36 1 4
3 Oct '20 men 3 David Warner 143 Jan 2019 - Sep 2020 33 1475 100 61.5 139. 128 39 2 16
4 Oct '20 men 4 Kamran Khan 135 Jan 2019 - Feb 2020 21 630 88 31.5 135. 50 39 0 5
5 Oct '20 men 5 Devdutt Padikkal 135 Nov 2019 - Sep 2020 15 691 122 57.6 167. 72 35 1 7
6 Oct '20 men 6 Devon Conway 121 Dec 2018 - Jan 2020 20 906 105 56.6 145. 113 19 2 5
7 Oct '20 men 7 Jos Buttler 121 Oct 2018 - Oct 2020 23 817 89 37.1 145. 93 32 0 8
8 Oct '20 men 8 Virat Kohli 119 Nov 2018 - Sep 2020 35 1174 100 40.5 141. 90 43 1 8
9 Oct '20 men 9 Kevin O'Brien 119 Oct 2018 - Sep 2020 38 1145 124 31.0 158. 107 59 1 5
10 Oct '20 men 10 Eoin Morgan 118 Oct 2018 - Oct 2020 34 1008 91 38.8 165. 69 66 0 8
# ... with 1,990 more rows
Case 3: all rounders
# all-rounders
future_imap_dfr(timeline("http://bigbashboard.com/rankings/bbl/all-rounders")[1:10], ~read_rankings(.x, .y))
# A tibble: 547 x 13
Date Title Rank Name Points Dates M R Ave SR W Econ Ave.1
<chr> <chr> <chr> <chr> <chr> <chr> <int> <int> <dbl> <dbl> <int> <dbl> <dbl>
1 8 Feb '20 men 1 D'Arcy Short 70 22 Dec 2018 - 30 Jan 2020 24 994 49.7 137. 16 8.61 29.1
2 8 Feb '20 men 2 Travis Head 49 11 Jan 2019 - 01 Feb 2020 11 291 29.1 132. 4 8.08 24.2
3 8 Feb '20 men 3 Mohammad Nabi 40 20 Dec 2018 - 27 Jan 2020 20 388 29.8 129. 13 7.9 30.4
4 8 Feb '20 men 4 Chris Morris 38 21 Dec 2019 - 06 Feb 2020 15 112 12.4 147. 22 8.01 19.4
5 8 Feb '20 men 5 Glenn Maxwell 37 21 Dec 2018 - 08 Feb 2020 30 729 36.4 146. 13 7.36 31.2
6 8 Feb '20 men 6 Daniel Sams 35 21 Dec 2018 - 06 Feb 2020 31 230 9.2 119. 45 8.19 17.3
7 8 Feb '20 men 7 Ben Cutting 33 19 Dec 2018 - 27 Jan 2020 28 466 24.5 137. 23 8.92 27.5
8 8 Feb '20 men 8 Mitchell Marsh 28 20 Dec 2018 - 26 Jan 2020 21 504 31.5 132. 6 9.56 43
9 8 Feb '20 men 9 Daniel Christian 27 20 Dec 2018 - 27 Jan 2020 30 382 21.2 124. 20 8.02 27.2
10 8 Feb '20 men 10 Rashid Khan 26 19 Dec 2018 - 01 Feb 2020 29 217 14.5 158. 38 6.65 19.5
# ... with 537 more rows
Q&A
How does the date work?
The new code scrapes both link and date from the same timeline on the website. Link is that href attribute; date is the text. See that timeline function. In this way, I avoid using URL to get the date.
Why did I encounter this Error: Can't recycle 'Date' (size 200) to match '..3' (size 190)?
Because there are tables as follows (also see this link)
which differs from your description that the ranking and stats tables always have the same number of rows.
这篇关于如何从 URL 中只有月份和年份的网站中的多个表中提取内容的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
