使用 R 和 rvest/httr 进行登录和重定向的网页抓取 [英] Web-Scraping with Login and Redirect using R and rvest/httr

问题描述

我想从网页中抓取信息.有一个登录屏幕，当我登录时，我可以访问所有类型的页面，我想从中获取信息(例如玩家的姓氏，对象 .lastName).我正在使用 R 和包 rvest 和 httr.

I would like to scrape information from a webpage. There is a login screen, and when I am logged in, I can access all kinds off pages from which I would like to scrape information (such as the last name of a player, the object .lastName). I am using R and the packages rvest and httr.

不知何故，登录似乎有效，但我不知道如何重定向到我需要从中获取信息的页面.

Somehow, the login seems to work, but I am clueless how to be redirected to the page I need to get the info from.

登录表单可以在 http://kickbase.sky.de/anmelden 访问，相关页面的格式为 http://kickbase.sky.de/spielerprofil/玩家姓名/号码，例如http://kickbase.sky.de/spielerprofil/nadiem-amiri/1639#.

The login form can be accessed on http://kickbase.sky.de/anmelden and the relevant pages have the form http://kickbase.sky.de/spielerprofil/player-name/number, e.g. http://kickbase.sky.de/spielerprofil/nadiem-amiri/1639#.

这是我使用的代码.非常感谢您的帮助.

Here is the code I used. Thank you very much for your help.

install.packages("rvest")
install.packages("httr")
library(rvest)
library(httr)

handle <- handle("http://kickbase.sky.de")  # Create handle
path   <- "anmelden" #  Login Path

# fields found in the login form.
login <- list(
  email = "[e-mail]"
  ,password  = "tester"
  ,redirect_url =  # I want to be redirected to this page and then scrape info from here
    "http://kickbase.sky.de/spielerprofil/nadiem-amiri/1639#"
)

response <- POST(handle = handle, path = path, body = login)

webpage <- read_html(response)
name_data <- html_text(html_nodes(webpage, ".lastName"))
name_data

推荐答案

library(rvest)
url<-"https://kickbase.sky.de/"
page<-html_session(url)
page<-rvest:::request_POST(page,url="https://kickbase.sky.de/api/v1/user/login",
                           body=list("email"="[e-mail]",
                                     "password"="tester",
                                     "redirect_url"="http://kickbase.sky.de/spielerprofil/nadiem-amiri/1639#"),
                         encode='json'
                           )
player_page<-jump_to(page,"https://kickbase.sky.de/api/v1/news?skip=0&player=1639&limit=3")
data<-jsonlite::fromJSON(readBin(player_page$response$content,what="json"))

print(data)

请注意，该网站提供了一个 API，您可以在此处获取数据https://kickbase.sky.de/api/v1/news?skip=0&player=1639&limit=3

Please note that the website provides an API and that is where you get the data https://kickbase.sky.de/api/v1/news?skip=0&player=1639&limit=3

变量 data 包含所有需要的信息

variable data has all the information needed

这篇关于使用 R 和 rvest/httr 进行登录和重定向的网页抓取的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！