单击后正在抓取.aspx站点 [英] Scraping .aspx site after click

问题描述

我正在尝试从以下位置收集我的中队的日程安排数据： https://www.cnatra.navy.mil/scheds/schedule_data.aspx?sq=vt-9

我已经知道如何使用BeautifulSoup提取数据，方法如下：

import urllib2
from urllib2 import urlopen
import bs4 as bs

url = 'https://www.cnatra.navy.mil/scheds/schedule_data.aspx?sq=vt-9'
html = urllib2.urlopen(url).read()
soup = bs.BeautifulSoup(html, 'lxml')
table = soup.find('table')
print(table.text)

但是，表格隐藏在所选日期下(如果不是当天)，并按下了"查看日程"按钮。

我如何修改我的代码以‘按下’查看计划‘按钮，这样我就可以抓取数据了？如果代码还可以选择日期，则会加分！

我尝试使用：

import urllib2
from urllib2 import urlopen
import bs4 as bs
from selenium import webdriver

driver = webdriver.Chrome("/users/base/Downloads/chromedriver")
driver.get("https://www.cnatra.navy.mil/scheds/schedule_data.aspx?sq=vt-9")
button = driver.find_element_by_id('btnViewSched')
button.click()

它成功打开Chrome并‘点击’该按钮，但我无法从中获取，因为地址未更改。

推荐答案

您可以使用PUREselenium获取时间表：

from selenium import webdriver

driver = webdriver.Chrome('chromedriver.exe')
driver.get("https://www.cnatra.navy.mil/scheds/schedule_data.aspx?sq=vt-9")
button = driver.find_element_by_id('btnViewSched')
button.click()
print(driver.find_element_by_id('dgEvents').text)

输出：

TYPE VT Brief EDT RTB Instructor Student Event Hrs Remarks Location
Flight VT-9 07:45 09:45 11:15 JARVIS, GRANT M [LT] LENNOX, KEVIN I [ENS] BI4101 1.5 2 HR BRIEF MASS BRIEF  
Flight VT-9 07:45 09:45 11:15 MOYNAHAN, WILLIAM P [CDR] FINNERAN, MATTHEW P [1stLt] BI4101 1.5 2 HR BRIEF MASS BRIEF  
Flight VT-9 07:45 12:15 13:45 JARVIS, GRANT M [LT] TAYLOR, ADAM R [1stLt] BI4101 1.5 2 HR BRIEF MASS BRIEF @ 0745 W/ JARVIS MEI OPS  
Flight VT-9 07:45 12:15 13:45 MOYNAHAN, WILLIAM P [CDR] LOW, TRENTON G [ENS] BI4101 1.5 2 HR BRIEF MASS BRIEF @ 0745 W/ MOYNAHAN MEI OPS  
Watch VT-9   00:00 14:00 ANDERSON, LAURA [LT]   ODO (ON CALL) 14.0    
Watch VT-9   00:00 14:00 ANDERSON, LAURA [LT]   ODO (ON CALL) 14.0    
Watch VT-9   00:00 23:59 ANDERSON, LAURA [LT]   ODO (ON CALL) 24.0    
Watch VT-9   00:00 23:59 ANDERSON, LAURA [LT]   ODO (ON CALL) 24.0    
Watch VT-9   07:00 19:00   STUY, JOHN [LTJG] DAY IWO 12.0    
Watch VT-9   19:00 07:00   STRACHAN, ALLYSON [LTJG] IWO 12.0    

这篇关于单击后正在抓取.aspx站点的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！