使用rvest抓取时如何重用会话以避免重复登录? [英] how to reuse a session to avoid repeated login when scraping with rvest?

问题描述

我开发了一些代码来抓取基于 这个话题.登录后我需要抓取很多页面，但现在我的代码似乎为每个网址重复登录网站.我如何重用"会话以避免重复登录，以便代码可以运行得更快?伪代码如下:

I developed some codes to scraping traffic data based this topic.I need to scrape many pages after log in, but right now my codes seem repeatedly log in the site for each url. How can I ‘reuse’ the session to avoid repeated log in so that, hopefully, the codes can run faster? Here's the pseudo-code:

generateURL <- function(siteID){return siteURL}

scrapeContent <- function(siteURL, session, filled_form){return content}

mainPageURL <- 'http://pems.dot.ca.gov/'
pgsession <- html_session(mainPageURL)
pgform <- html_form(pgsession)[[1]]
filled_form <- set_value(pgform, 'username'='myUserName', 'password'='myPW')

siteIDList = c(1,2,3)
vectorOfContent <- vector(mode='list', length=3) #to store all the content

i=1
for (siteID in siteIDList){
    url = generateURL(siteID)
    content = scrapeContent(url, pgsession, filled_form)
    vectorOfContent[[i]]=content
    i = i +1}

我阅读了 rvest 文档，但其中没有此类详细信息.我的问题:如何重用"会话以避免重复登录?谢谢！

I read the rvest documnentation but there is no such details in it. My question: How can I ‘reuse’ the session to avoid repeated log in? Thanks!

推荐答案

你可以这样做:

require(rvest)
pgsession <- html_session(mainPageURL)
pgform <- html_form(pgsession)[[1]]
filled_form <- set_value(pgform, 'username'='myUserName', 'password'='myPW')
s <- submit_form(pgsession, pgform) # s is your logged in session

vectorOfContent <- vector(mode='list', length=3)

for (siteID in siteIDList){
  url <- generateURL(siteID)
  # jump_to navigates within the session, read_html parses the html
  vectorOfContent[[siteID]]=s %>% jump_to(generateURL) %>% read_html()
  }

这篇关于使用rvest抓取时如何重用会话以避免重复登录?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！