在 R 中抓取相关页面 [英] Scraping related pages in R

问题描述

我正在尝试从几个姐妹 URL 中抓取数据进行分析.以前的线程 Scraping网页、页面上的链接以及使用 R 形成表格有助于使用以下脚本使我走上正确的道路:

I am trying to scrape data from several sister URLs for analysis. A previous thread Scraping a web page, links on a page, and forming a table with R was helpful in getting me on the right path with the following script:

rm(list=ls())
library(XML)
library(RCurl) 

#=======2013========================================================================
url2013 = 'http://www.who.int/csr/don/archive/year/2013/en/index.html'
doc <- htmlParse(url2013)
dummy2013 <- data.frame(
  dates = xpathSApply(doc, '//*[@class="auto_archive"]/li/a', xmlValue),
  hrefs = xpathSApply(doc, '//*[@class="auto_archive"]/li/a', xmlGetAttr,'href'),
  title = xpathSApply(doc, '//*[@class="link_info"]/text()',  xmlValue)
)

dummy2013$text = unlist(lapply(dummy2013$hrefs,function(x)
{
  url.story <- gsub('/entity','http://www.who.int',x)
  texts <- xpathSApply(htmlParse(url.story), 
                       '//*[@id="primary"]',xmlValue)
}))

dummy2013$link <- gsub('/entity','http://www.who.int',dummy2013$hrefs)

write.csv(dummy2013, "whoDON2013.csv")

然而，应用于姐妹 URL 时，事情就坏了.尝试

However, applied to sister URLs, things break. Trying

#=======2011========================================================================
url2011 = 'http://www.who.int/csr/don/archive/year/2011/en/index.html'
doc <- htmlParse(url2011)
dummy2011 <- data.frame(
  dates = xpathSApply(doc, '//*[@class="auto_archive"]/li/a', xmlValue),
  hrefs = xpathSApply(doc, '//*[@class="auto_archive"]/li/a', xmlGetAttr,'href'),
  title = xpathSApply(doc, '//*[@class="link_info"]/text()',  xmlValue)
)

例如，产生

## Error in data.frame(dates = xpathSApply(doc, "//*[@class=\"auto_archive\"]/li/a",  : 
  arguments imply differing number of rows: 59, 60

http://www 出现类似错误.who.int/csr/don/archive/year/2008/en/index.html 和 http://www.who.int/csr/don/archive/year/2006/en/index.html.我不擅长 HTML 或 XML；任何想法表示赞赏.

Similar errors occur for http://www.who.int/csr/don/archive/year/2008/en/index.html and http://www.who.int/csr/don/archive/year/2006/en/index.html. I'm not handy with HTML or XML; any ideas appreciated.

推荐答案

您可以先选择标题，然后找到与之关联的 href

You can select the titles first then find the href associated with them

require(XML)
url2011 = 'http://www.who.int/csr/don/archive/year/2011/en/index.html'
doc <- htmlParse(url2011)
titleNodes <- getNodeSet(doc, '//*[@class="link_info"]')
hrefNodes <- sapply(titleNodes, getNodeSet, path = './preceding-sibling::a')

dummy2011 <- data.frame(
    dates = sapply(hrefNodes, xmlValue),
    hrefs = sapply(hrefNodes, xmlAttrs),
    title = sapply(titleNodes, xmlValue),
    stringsAsFactors = FALSE
)

更新:

删除可以使用的重复值

dummy2011 <- dummy2011[!duplicated(dummy2011$hrefs),]

这篇关于在 R 中抓取相关页面的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！