使用R通过链接名称获取下载URL [英] Using R to get download URL by link name
本文介绍了使用R通过链接名称获取下载URL的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我正尝试使用 rvest 从这个网站。文件名是固定的，但是下载网址不符合模式（只有几十个数字），所以我无法根据任何标准构建下载网址列表。我如何使用链接名称下载实际文件？
到目前为止，我可以获得感兴趣的文件列表（基于CSS选择器），并且可以获取网站上所有链接的列表，但我不知道如何匹配它们。我需要能够检查网站的变化并下载任何名称更改的文件，因此使用文件名访问文件非常重要。我对HTML / CSS并不是很熟悉，所以这可能就是为什么我无法弄清这个可能简单的任务。
library（rvest）
＃带有下载文件列表的网址
url< - http://www-air.larc.nasa.gov/cgi- bin / ArcView / actamerica.2016？C130 = 1
doc< - read_html（url）
＃获取CSS选择器中的所有内容td a
所有< - html_text（html_nodes（doc，td a））
＃获取某些文件名的列表
filetype< - PICARRO
文件< - all [grep（filetype，all）]
＃这将返回页面上所有链接的列表，
＃但我不确定如何匹配链接与他们的名字
html_attr（html_nodes（doc，a），href）
预先感谢您的帮助。
解决方案
获取所有可下载的文件名和URL ：
library（httr）
library（rvest）
library（tidyverse）
pg < - read_html（http://www-air.larc.nasa.gov/cgi-bin/ArcView/actamerica.2016?C130=1）
fils< - html_nodes（pg，xpath =.// a [contains（@href，'cgi-bin / enzFile'）]）
data_frame（
filename = html_text（fils），
link = sprintf（http：//www-air.larc.nasa.gov%s，html_attr（fils，href））
） - > xdf
glimpse（xdf）
##观察：719
##变量：2
## $文件名< chr> ACTAMERICA-Elevation_C130_20160711_R0.ict，ACTAMERICA-Elevation_C130_20160715_R0.ict，ACTAMERI ...
## $ link http://www-air.larc.nasa.gov/cgi- bin / enzFile？f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f ...
xdf
##＃A tibble：719 x 2
## filename link
##< chr> < CHR>
## 1 ACTAMERICA-Elevation_C130_20160711_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303731315f52302e696374
## 2 ACTAMERICA-Elevation_C130_20160715_R0.ict HTTP：// WWW -air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303731355f52302e696374
## 3 ACTAMERICA-Elevation_C130_20160718_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303731385f52302e696374
## 4 ACTAMERICA-Elevation_ C130_20160719_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303731395f52302e696374
## 5 ACTAMERICA-Elevation_C130_20160721_R0.ict http://www-air.larc.nasa.gov/的cgi-bin / enzFile？f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303732315f52302e696374
## 6 ACTAMERICA-Elevation_C130_20160722_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303732325f52302e696374
## 7 ACTAMERICA，Elevation_C130_20160725_R0 .ict http：// www -air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303732355f52302e696374
## 8 ACTAMERICA-Elevation_C130_20160726_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303732365f52302e696374
## 9 ACTAMERICA-Elevation_C130_20160727_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303732375f52302e696374
## 10 ACTAMERICA-Elevation_C130_20160801_R0.ict HTTP：// WWW- air.larc.nasa.gov/cgi-bin/enz文件？f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303830315f52302e696374
##＃...有709多个行
获取那些你注意：
$ b $ pre $ picarro< - filter（xdf，grepl（PICARRO，filename））
下载它们：
walk2（picarro $链接，picarro $文件名，download.file）
##试图URL 'http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d5049434152524f5f433133305f32303136303532375f52422e696374'
＃ ＃内容类型'text / plain'长度1023662字节（999 KB）
## ============================= =====================
##已下载999 KB
##
##试图URL 'http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d5049434152524f5f433133305f32303136303731315f52302e696374'
##内容类型 '文本/纯' 长度886392个字节（865 KB）
＃ ＃================================================= =
##下载865 KB
##
##试图URL 'http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d5049434152524f5f433133305f32303136303731355f52302e696374'
##内容类型'text / plain'长度530339字节（517 KB）
## =========================== =======================
##已下载517 KB
等。
I'm trying to use rvest to download a list of files from this site. The file names are regular, but the download URLs don't match a pattern (just dozens of digits), so I can't construct a list of download URLs based on any criteria. How can I use link names to download the actual files?
So far, I can get a list of the files of interest (based on CSS selector), and I can get a list of all the links on the site, but I'm not sure how to match them up. I'll need to be able to check the site for changes and download any files with changed names, so using the file name to access the file is important. I'm not very familiar with HTML/CSS, so that might be why I can't figure out this possibly-simple task.
library(rvest)
# url with list of download files
url <- "http://www-air.larc.nasa.gov/cgi-bin/ArcView/actamerica.2016?C130=1"
doc <- read_html(url)
# getting everything within the CSS selector "td a"
all <- html_text(html_nodes(doc, "td a"))
# getting list of certain file names
filetype <- "PICARRO"
files <- all[grep(filetype, all)]
# this returns a list of all links on the page,
# but I'm not sure how to match the links up with their names
html_attr(html_nodes(doc, "a"), "href")
Thank you in advance for any help.
解决方案
Slightly different approach.
Grab all downloadable filenames and URLs:
library(httr)
library(rvest)
library(tidyverse)
pg <- read_html("http://www-air.larc.nasa.gov/cgi-bin/ArcView/actamerica.2016?C130=1")
fils <- html_nodes(pg, xpath=".//a[contains(@href, 'cgi-bin/enzFile')]")
data_frame(
filename = html_text(fils),
link = sprintf("http://www-air.larc.nasa.gov%s", html_attr(fils, "href"))
) -> xdf
glimpse(xdf)
## Observations: 719
## Variables: 2
## $ filename <chr> "ACTAMERICA-Elevation_C130_20160711_R0.ict", "ACTAMERICA-Elevation_C130_20160715_R0.ict", "ACTAMERI...
## $ link <chr> "http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f...
xdf
## # A tibble: 719 x 2
## filename link
## <chr> <chr>
## 1 ACTAMERICA-Elevation_C130_20160711_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303731315f52302e696374
## 2 ACTAMERICA-Elevation_C130_20160715_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303731355f52302e696374
## 3 ACTAMERICA-Elevation_C130_20160718_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303731385f52302e696374
## 4 ACTAMERICA-Elevation_C130_20160719_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303731395f52302e696374
## 5 ACTAMERICA-Elevation_C130_20160721_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303732315f52302e696374
## 6 ACTAMERICA-Elevation_C130_20160722_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303732325f52302e696374
## 7 ACTAMERICA-Elevation_C130_20160725_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303732355f52302e696374
## 8 ACTAMERICA-Elevation_C130_20160726_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303732365f52302e696374
## 9 ACTAMERICA-Elevation_C130_20160727_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303732375f52302e696374
## 10 ACTAMERICA-Elevation_C130_20160801_R0.ict http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d456c65766174696f6e5f433133305f32303136303830315f52302e696374
## # ... with 709 more rows
Get the ones you care about:
picarro <- filter(xdf, grepl("PICARRO", filename))
Download them:
walk2(picarro$link, picarro$filename, download.file)
## trying URL 'http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d5049434152524f5f433133305f32303136303532375f52422e696374'
## Content type 'text/plain' length 1023662 bytes (999 KB)
## ==================================================
## downloaded 999 KB
##
## trying URL 'http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d5049434152524f5f433133305f32303136303731315f52302e696374'
## Content type 'text/plain' length 886392 bytes (865 KB)
## ==================================================
## downloaded 865 KB
##
## trying URL 'http://www-air.larc.nasa.gov/cgi-bin/enzFile?f49DA0512C4E81E3C01FDB44A33CD88AAFE2f7075622d6169722f414354414d45524943412f323031362f433133305f41495243524146542f444947414e47492e4a4f534855412f414354414d45524943412d5049434152524f5f433133305f32303136303731355f52302e696374'
## Content type 'text/plain' length 530339 bytes (517 KB)
## ==================================================
## downloaded 517 KB
etc.
这篇关于使用R通过链接名称获取下载URL的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
