如何在R中刮取JSP页面? [英] How to scrape a JSP page in R?
问题描述
我想在R中抓取以下页面的内容: http://directoriosancionados.funcionpublica.gob.mx/SanFicTec/jsp/Ficha_Tecnica/SancionadosN.htm
I want to scrape the content of the following page in R: http://directoriosancionados.funcionpublica.gob.mx/SanFicTec/jsp/Ficha_Tecnica/SancionadosN.htm
但是，我找不到任何可以帮助我获取信息的HTML标记或其他工具.
However, I'm not able to locate any HTML tag or any other tool that could help me to obtain the information.
我感兴趣的是使用"INHABILITADOS Y MULTADOS"部分的信息来构建数据框.就像下面的图片一样:
I'm interested in build a data frame with the information of the section "INHABILITADOS Y MULTADOS" like in the following images:
选择此选项后，将出现一个包含多个提供程序的菜单，每个菜单都有一个包含我要重新收集的信息的特定表格.
When this option is selected, a menu of several providers appears, each one with a particular table with the information I want to recollect.
推荐答案
通常，您可以对请求使用GET方法.但是对于该网站，您需要使用POST方法:
Normally, you can use GET method for requests. But for that website,you need to use POST method:
在chrome开发人员模式下检查网络标签(按F12键)
在以下图像中，在POST请求的正文中提交表单数据.
在onclick中查找样式:onlick值用于提交表单
Find patterns in onclick: the onlick value is used for submitting forms
以下脚本应该起作用:
library(httr)
library(rvest)
library(stringr)
library(dplyr)
my_url <- "http://directoriosancionados.funcionpublica.gob.mx/SanFicTec/jsp/Ficha_Tecnica/SancionadosN.jsp"
my_ua <- "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36"
#use post method instead of get to get correct response
response <- POST(my_url,
user_agent(my_ua),
body = list(cmdsan = "INHABILITA",
tipoqry = "INHABILITA",
mostrar_msg = "SI"),
encode = "form")
href_nodes <- content(response) %>%
html_node("table") %>%
html_nodes("a")
link_text <- href_nodes %>%
html_text() %>%
as.tibble() %>%
rename(text = value)
form_items <- href_nodes %>%
html_attr("onclick") %>% # extract items to POST
str_extract("(?<=\\().*?(?=\\))") %>% # extract everything inside brackets
str_split("\\,",simplify = T) %>%# split POST items
as.tibble() %>%
mutate(V1 = str_sub(V1,start = 2,end =-2))
submit_table <- bind_cols(link_text,form_items)
#using POST method to get to the page you want
#for example, if you want to go to page A Y M CONSTRUCTORA, S.A. DE C.V (row 2)
#you should:
row_num <- 2
my_url2 <- "http://directoriosancionados.funcionpublica.gob.mx/SanFicTec/jsp/Ficha_Tecnica/FichaSinTabla.jsp"
response1 <- POST(my_url2,
user_agent(my_ua),
body = list(expe = submit_table$V1[row_num],
tipo = submit_table$V2[row_num],
persona = submit_table$V3[row_num]),
encode = "form")
内容，稍后将用于发出POST请求以获取每个单独页面中的内容.
content in submit_table, which will be used later to make POST request to get content in each individual page.
> submit_table
# A tibble: 1,329 x 4
text V1 V2 V3
<chr> <chr> <chr> <chr>
1 A AND P INTERNATIONAL 185770002/2016 1 3
2 A Y M CONSTRUCTORA, S.A. DE C.V. 000090121/2006 1 3
3 A Y V INDUSTRIAL Y COMERCIAL, S.A. DE C.V. 184000001/2013 1 3
4 A+D ARQUITECTOS, S.A. DE C.V. 181640187/2006 1 3
5 A.D.C. Consultores y Servicios, S.A de C.V. 111510007/2005 1 3
6 AARON VERA MORALES 006410056/2011 1 3
7 ABASTECEDORA DE FÁRMACOS, S.A. DE C.V. 006410002/2014 1 3
8 ABASTECEDORA EZCO, S.A. DE C.V. 000070024/2016 1 3
9 ABEL ZURITA MAYO 000200012/2014 1 3
10 ABS TECNOLOGÍA, S.A. DE C.V. 090850001/2016 1 3
# ... with 1,319 more rows
您可以使用rvest中的函数使用响应来提取这些元素:
(content(response1) %>% html_nodes(".normal") %>% html_text() %>% str_trim())[3]
将返回:
[1] "Publicación en el DOF: 05 DE ABRIL DE 2007Monto de la Multa: $ 72,540.00Plazo de inhabilitación: 3 MESESInicia: 06 DE ABRIL DE 2007Termina: 06 DE JULIO DE 2007"
这篇关于如何在R中刮取JSP页面?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
